A method for repairing etcd database files

Abstract

The present invention discloses a method for repairing etcd database files, which relates to the technical field of data repair. Based on the data storage mechanism of boltDB, the underlying data storage engine used by etcd, it scans and parses the data content stored in the data db file page by page, classifies and stores it according to page types, parses the data in all leafPages in the mvcc format in etcd, and inserts the parsed data content into a new db file initialized in the target database, so as to realize the repair of the damaged data db file. The present invention can realize the repair of damaged data without backup data or during the backup period for scenarios where the db file structure is complete but the metadata is damaged, and scenarios where both the db file structure and metadata are damaged.

Description

Technical Field

[0001] The present invention relates to the technical field of data repair, and specifically, it is a method for repairing etcd database files. Background Art

[0002] Etcd is a distributed key-value storage system used for sharing configurations, service discovery, and multi-version key-value data storage. It is written in the Go language and has excellent cross-platform support. The member nodes maintain strong consistency through the Raft algorithm. As one of the most popular cloud-native storages, etcd is widely used in the field of cloud computing. The open-source container orchestration system Kubernetes uses etcd to store various resource objects.

[0003] As the database storage technology used by the kubernetes container orchestration engine, with the popularization of cloud computing and cloud-native technologies, etcd has more and more usage scenarios. How to ensure the security of etcd data and how to deal with data failures are issues that users need to consider. Etcd provides professional tools for snapshot backup of the database. How to perform database backup and how long the backup interval should be also need to be comprehensively considered and weighed by users according to the existing cluster business usage to formulate a backup plan. If no data backup is taken or the backup interval is too long, data loss may occur during a failure, and sometimes the losses are irreparable and immeasurable. Summary of the Invention

[0004] In view of the current technical development needs and deficiencies, the present invention provides a method for repairing etcd database files to ensure the repair of damaged data without backup data or during the backup period.

[0005] The technical solution adopted by the method for repairing etcd database files of the present invention to solve the above technical problems is as follows:

[0006] A method for repairing etcd database files, based on the data storage mechanism of the underlying data storage engine boltDB used by etcd, scans and analyzes the data content stored in the data db file page by page, classifies and stores it according to the page type, analyzes the data in all leafPages in the mvcc format of etcd, and inserts the parsed data content into a new db file initialized in the target database to achieve the repair of the damaged data db file.

[0007] For the above method, the page types of the boltDB data storage engine include: metaPage, branchPage, leafPage, and freelistPage, where:

[0008] metaPage is used to store the total number of pages of the db file, the current transaction number, and PageSize, distributed on page 0 and page 1;

[0009] branchPage is used to store the leafPage information contained in the current branch;

[0010] LeafPage corresponds to the leaf node in the B+ tree and is used to store all key-value pairs and also to store the subbucket information of the bucket. When organizing data, LeafPage stores the element header and the element itself separately. The element header data is fixed-length, and the corresponding element can be quickly accessed by pressing the index.

[0011] FreelistPage is used to store one or more consecutive pages in the db file, and is used to save the id list information of pages released due to modification operations during the use of the db.

[0012] Specifically, based on the type of damage to the database file, there are two repair methods:

[0013] (1) In the scenario where the database file structure is intact but the metadata is damaged, repair the total number of pages in the root bucket of the database file meta page;

[0014] (2) In the scenario where both the database file structure and metadata are damaged, the damaged database file will be completely migrated.

[0015] Furthermore, in the scenario where the db file structure is intact but the metadata is damaged, the db file size is inconsistent with the meta page. In this case, the total number of pages needs to be recalculated based on the db file size and the operating system PageSize, and the total number of pages metadata in the meta page of the damaged db file needs to be corrected and records whose freelist page numbers exceed the total number of pages need to be removed.

[0016] Alternatively, in the scenario where the db file structure is intact but the metadata is damaged, the specific db file repair process is as follows:

[0017] (1) Parse the db file page PageSize and obtain the PageSize size, including:

[0018] (1.1) Read the db file starting from 0 and not less than 28 bytes of content, and parse the operating system PageSize information stored in the db file according to the boltDB storage engine metaPage page structure.

[0019] (1.2) If the PageSize can be parsed normally in step (1.1), skip this step. Execute the getconf PAGESIZE command on the server where the etcd cluster is running, and record the returned result as the PageSize;

[0020] (2) Calculate the total number of pages of the db file

[0021] (2.1) Obtain the total number of bytes of the db file: On the server where the db file is located, obtain the total number of bytes of the file by means of the command for viewing file information, or obtain the total number of bytes of the db file through program coding.

[0022] (2.2) Divide the total number of bytes obtained in step (2.1) by the PageSize obtained in step (1), and then add 1 to the result, which is recorded as the actual total number of pages of the db file;

[0023] (3) Compare the metadata of the db file

[0024] (3.1) According to the PageSize obtained in step (1), read the content of one PageSize byte of the db file, and parse the pgid in the db file according to the metaPage page structure of the boltDB storage engine.

[0025] (3.2) If the pgid calculated in step (3.1) is inconsistent with the total number of pages calculated in step (2.2), then correct the pgid in the metaPage of the db file to the total number of pages calculated in step (2.2);

[0026] (4) Correct the redundant check and freelist overflow pages in the metaPage of the db file

[0027] (4.1) According to the redundant check algorithm of the metaPage page of the boltDB storage engine, recalculate the check fingerprint and correct the check fingerprint in the metaPage of the db file.

[0028] (4.2) Traverse the freelist page, remove the records whose pgid is greater than the total number of pages calculated in step (2.2), and correct the content of the freelist in the db file.

[0029] Further optionally, in the scenario where both the db file structure and metadata are damaged, perform a complete data migration on the damaged db file. The specific process is as follows:

[0030] (1) Parse the data of the db file page by page, specifically including:

[0031] (1.1) According to the PageSize pagination size of the operating system, read the content of the db file page by page and parse it into the data structure in the boltDB storage engine.

[0032] (1.2) Store them into different structures respectively according to the page type;

[0033] (2) Parse the elements in leafPage: Traverse all the pages in leafPage, eliminate the incomplete data records, and store the parsed key-value pairs into the map structure;

[0034] (3) Initialize a new db file in the target database, traverse the map structure of key-value pairs mentioned in step (2), and insert the key-value pairs into the new db file in sequence;

[0035] (4) Complete the cluster member information

[0036] (4.1) Insert the members bucket into the new db file,

[0037] (4.2) Insert the cluster member information into the members bucket of the new db file in sequence.

[0038] A method for repairing etcd database files according to the present invention has the beneficial effects compared with the prior art as follows:

[0039] The present invention can repair damaged database files, ensuring the repair of damaged data during the period without backup data or backup cycle. Brief Description of the Drawings

[0040] Attached Figure 1 is the repair process of the db file in the scenario where the db file structure is complete and the metadata is damaged according to the present invention;

[0041] Attached Figure 2 is the repair process of the db file in the scenario where both the db file structure and metadata are damaged according to the present invention. Detailed Embodiments

[0042] To make the technical solutions, the technical problems to be solved and the technical effects of the present invention clearer and more understandable, the following combines specific embodiments to clearly and completely describe the technical solutions of the present invention.

[0043] Embodiment 1:

[0044] This embodiment proposes a method for repairing etcd database files. Based on the data storage mechanism of the underlying data storage engine boltDB used by etcd, it scans and analyzes the data content stored in the data db file page by page, classifies and stores it according to the page type, analyzes all the data in leafPage in the mvcc format of etcd, and inserts the parsed data content into the new db file initialized in the target database to repair the damaged data db file.

[0045] BoltDB data storage engine page types include: metaPage, branchPage, leafPage, and freelistPage.

[0046] metaPage is used to store the total number of pages of the db file, the current transaction number, and PageSize, distributed on page 0 and page 1;

[0047] branchPage is used to store the leafPage information contained in the current branch;

[0048] LeafPage corresponds to the leaf node in the B+ tree and is used to store all key-value pairs and also to store the subbucket information of the bucket. When organizing data, LeafPage stores the element header and the element itself separately. The element header data is fixed-length, and the corresponding element can be quickly accessed by pressing the index.

[0049] FreelistPage is used to store one or more consecutive pages in the db file, and is used to save the id list information of pages released due to modification operations during the use of the db.

[0050] The method described in this embodiment includes the following two repair methods based on the damage type of the db file:

[0051] (1) In the scenario where the database file structure is intact but the metadata is damaged, repair the total number of pages in the root bucket of the database file meta page;

[0052] (2) In the scenario where both the database file structure and metadata are damaged, the damaged database file will be completely migrated.

[0053] Combined with attachment Figure 1 For (1), when the db file structure is intact but the metadata is damaged, the db file size is inconsistent with the meta page. In this case, the total number of pages must be recalculated based on the db file size and the operating system's PageSize. The total number of pages in the meta page of the damaged db file must be corrected, and records with freelist page numbers exceeding the total number of pages must be removed. The specific db file repair process is as follows:

[0054] (1) Parse the db file page PageSize and obtain the PageSize size, including:

[0055] (1.1) Read the db file starting from 0 and not less than 28 bytes of content, and parse the operating system PageSize information stored in the db file according to the boltDB storage engine metaPage page structure.

[0056] (1.2) If the PageSize can be parsed normally in step (1.1), skip this step and execute the getconf PAGESIZE command on the server running the etcd cluster. The returned result is recorded as the PageSize.

[0057] (2) Calculate the total number of pages in the db file

[0058] (2.1) Get the total number of bytes of the db file: Use the View File Information command on the server where the db file is located to get the total number of bytes of the file, or use program coding to get the total number of bytes of the db file.

[0059] (2.2) Divide the total number of bytes obtained in step (2.1) by the PageSize obtained in step (1), and add 1 to the result, which is the actual total number of pages in the db file;

[0060] (3) Compare db file metadata

[0061] (3.1) Get the PageSize size according to step (1), read the PageSize bytes of the db file, and parse the pgid in the db file according to the metaPage page structure of the boltDB storage engine.

[0062] (3.2) If the pgid calculated in step (3.1) is inconsistent with the total number of pages calculated in step (2.2), then correct the pgid in the metaPage of the db file to the total number of pages calculated in step (2.2);

[0063] (4) Corrected the redundancy check and freelist overflow pages in the db file metaPage

[0064] (4.1) According to the boltDB storage engine metaPage page redundancy check algorithm, recalculate the check fingerprint and correct the check fingerprint in the metaPage in the db file.

[0065] (4.2) Traverse the freelist pages, remove the records whose pgid is greater than the total number of pages calculated in step (2.2), and correct the freelist content of the db file.

[0066] Combined with attachment Figure 2 For (2), when both the db file structure and metadata are damaged, the damaged db file will be completely migrated. The specific process is as follows:

[0067] (1) Parse the db file data page by page, including:

[0068] (1.1) Read the content of the db file page by page according to the page size of the operating system PageSize, and parse it into the data structure in the boltDB storage engine.

[0069] (1.2) Store them in different structures according to the page type.

[0070] (2) Parse the elements in the leafPage: Traverse all the pages in the leafPage, remove the incomplete data records, and store the parsed key-value pairs in the map structure.

[0071] (3) Initialize a new db file in the target database, traverse the map structure of the key-value pairs mentioned in step (2), and insert the key-value pairs into the new db file one by one.

[0072] (4) Complete the cluster member information

[0073] (4.1) Insert the members bucket into the new db file.

[0074] (4.2) Insert the cluster member information into the members bucket of the new db file in sequence.

[0075] In summary, by using a method for repairing etcd database files according to the present invention, it is possible to repair damaged database files and ensure the repair of damaged data during the period without backup data or backup cycle.

[0076] The above application of specific examples has elaborated in detail the principle and implementation manner of the present invention. These embodiments are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, those skilled in the art of the present technology field, without departing from the principle of the present invention, any improvements and modifications made to the present invention shall fall within the scope of patent protection of the present invention.

Claims

1. A method for repairing etcd database files, characterized in that The method is based on the data storage mechanism of boltDB, the underlying data storage engine used by etcd. It scans and parses the data content stored in the data db file page by page, classifies and stores it by page type, parses the data in all leaf pages according to the mvcc format in etcd, and inserts the parsed data content into the new db file initialized by the target database to repair the damaged data db file. In this process: BoltDB data storage engine page types include: metaPage, branchPage, leafPage, and freelistPage. MetaPage is used to store the total number of pages in the database file, the current transaction number, and the PageSize, and is distributed on pages 0 and 1. BranchPage is used to store the leafPage information contained in the current branch. LeafPage corresponds to the leaf node in the B+ tree and is used to store all key-value pairs and also to store the subbucket information of a bucket. When organizing data, leafPage stores the element header and the element itself separately. The element header data is fixed-length, and the corresponding element can be quickly accessed by pressing the index. FreelistPage is used to store one or more consecutive pages in the database file and is used to save the ID list information of pages released due to modification operations during the use of the database. Based on the damage type of the db file, there are two repair methods: (1) In the scenario where the database file structure is intact but the metadata is damaged, repair the total number of pages in the root bucket of the database file's meta page. In this case, recalculate the total number of pages based on the database file size and the operating system's PageSize, correct the total number of pages in the meta page of the damaged database file, and remove records whose freelist page numbers exceed the total number of pages. (2) In the scenario where both the database file structure and metadata are damaged, complete data migration of the damaged database file is performed. The specific process is as follows: (1) Parse the db file data page by page; (2) Parse the elements in leafPage: traverse all pages in leafPage, remove incomplete data records, and store the parsed key-value pairs in the map structure; (3) Initialize a new db file in the target database and insert the elements parsed in step (2) into the new db file; (4) Complete cluster member information: (4.1) Insert the members bucket into the new db file, (4.2) Insert cluster member information into the members bucket of the new db file one by one.

2. A method for repairing etcd database files according to claim 1, characterized in that: In the scenario where the db file structure is intact but the metadata is damaged, the specific process of repairing the db file is as follows: (1) Parse the db file page PageSize and obtain the PageSize size; (2) Calculate the total number of pages in the db file (2.1) Get the total number of bytes of the db file: Use the View File Information command on the server where the db file is located to get the total number of bytes of the file, or use program coding to get the total number of bytes of the db file. (2.2) Divide the total number of bytes obtained in step (2.1) by the PageSize obtained in step (1), and add 1 to the result, which is the actual total number of pages in the db file; (3) Compare db file metadata (3.1) According to step (1), get the PageSize size, read the PageSize bytes of the db file, and parse the pgid in the db file according to the boltDB storage engine metaPage page structure. (3.2) If the pgid calculated in step (3.1) is inconsistent with the total number of pages calculated in step (2.2), then correct the pgid in the metaPage of the db file to the total number of pages calculated in step (2.2); (4) Corrected the redundancy check and freelist overflow pages in the db file metaPage (4.1) Based on the boltDB storage engine metaPage page redundancy check algorithm, recalculate the check fingerprint and correct the check fingerprint in the metaPage in the db file. (4.2) Traverse the freelist pages, remove the records whose pgid is greater than the total number of pages calculated in step (2.2), and correct the freelist content of the db file.

3. A method for repairing etcd database files according to claim 2, characterized in that: In the scenario where the db file structure is intact but the metadata is damaged, execute step (1) to parse the db file page PageSize and obtain the PageSize, including: (1.1) Read the db file starting from 0 and not less than 28 bytes of content, and parse the operating system PageSize information stored in the db file according to the boltDB storage engine metaPage page structure. (1.2) If the PageSize can be parsed normally in step (1.1), skip this step and execute the getconf PAGESIZE command on the server running the etcd cluster. The returned result is recorded as the PageSize.

4. The etcd database file repair method according to claim 1, characterized in that: In the scenario where both the db file structure and metadata are damaged, execute the step (1) of the db file repair process to parse the db file data page by page, specifically including: (1.1) Read the db file content page by page according to the operating system's PageSize paging size and parse it into the data structure in the boltDB storage engine. (1.2) According to the page type, they are stored in different structures.

5. A method for repairing etcd database files according to claim 4, characterized in that: Execute step (3), initialize a new db file in the target database, traverse the key-value pair map structure mentioned in step (2), and insert the key-value pairs into the new db file one by one.

Images