A golden db database damage file row-level data recovery method and device

Abstract

The application discloses a GoldenDB database damage file row-level data recovery method and equipment, the method comprises the following steps: step S1: scanning the GoldenDB data file to be recovered, identifying and caching all data pages; step S2: based on the metadata of the data page, filtering out the cluster index leaf node page containing user data from the cached data page; step S3: for each leaf node page filtered out, analyzing its internal record organization structure, and performing row-level data recovery; the row-level data recovery comprises traversing user records according to virtual record pointers, and cross-checking and repairing the traversal path in combination with the page directory slot, so as to extract and mark user records one by one; step S4: outputting all recovered row-level data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a row-level data recovery method and device for a GoldenDB database damaged file. BACKGROUND

[0002] In a GoldenDB database, storage engines are widely used to persist and manage structured data. The independent table space mechanism provided by the storage engines allows the data of each table to be stored in an independent data file, thereby improving the management flexibility and physical isolation capability of the data. However, in actual application environments, due to hardware aging, storage medium defects, system abnormal shutdown, unexpected power failure, or even operating system level abnormalities, the storage files of the GoldenDB database can be damaged. Such damage can manifest as loss of file header information, page structure abnormalities, data block verification failures, and the like. In particular, due to disk medium silent damage, the failure cannot be detected in time at the file system or database layer, and when the database is running and the damaged page is accessed or verified, a consistency check failure is triggered, resulting in instance abnormal termination or service interruption.

[0003] Currently, the mainstream damaged data recovery tools are limited to page-level recovery. Due to the coarse recovery granularity, once a single data page is damaged, all the row records in the page will be lost, and finer-grained row-level repair cannot be performed. Therefore, an innovative data recovery mechanism is urgently needed to break through the granularity limitation of existing recovery tools, support quick positioning and extraction of row-level data stored in a partially damaged GoldenDB data file structure, and thereby effectively improve the data recoverability of the database in abnormal scenarios. SUMMARY

[0004] The present application provides a row-level data recovery method and device for a GoldenDB database damaged file to solve the above problems.

[0005] In one aspect, the present application provides a row-level data recovery method for a GoldenDB database damaged file, which comprises the following steps:

[0006] Step S1: scanning a GoldenDB data file to be recovered, identifying and caching all data pages;

[0007] Step S2: based on the metadata of the data pages, filtering out cluster index leaf node pages containing user data from the cached data pages;

[0008] Step S3: For each leaf node page screened out, parse its internal record organization structure, and perform row-level data recovery; the row-level data recovery includes traversing user records according to a virtual record pointer, and cross-checking and repairing a traversal path in combination with a page directory slot, so as to extract and mark user records one by one;

[0009] Step S4: Output all the recovered row-level data.

[0010] In an implementation manner of the present application, the step S1 further includes sequentially reading each data page in the data file, and performing integrity checking and attribute analysis on the data page, and the process specifically includes the following steps:

[0011] Calculate a checksum of the data page and compare it with a checksum stored in the data page, and verify consistency of log sequence numbers of a file header and a file tail, so as to determine whether the data page is damaged and cache the damaged page separately;

[0012] For the data page passing the integrity checking, parse a page type in a file header of the data page, and screen out the data page of the type of index page;

[0013] Parse page header information of the index page, and obtain an index ID and a level in a B+ tree of the index page;

[0014] Cache a leaf node page of which the index ID matches a target cluster index ID and the level is 0, so as to complete positioning and collection of an effective data bearing page.

[0015] In an implementation manner of the present application, the organization structure of the data page includes a file header at a page head, a page header for storing record management information in the page, virtual minimum and maximum records as traversal boundaries, a user record area for storing actual row data, a free space area for subsequent insertion, a page directory for quickly locating records, and a file tail at a page tail for integrity checking; the row-level data recovery mainly acts on the user record area, and relies on the page directory and the virtual record for navigation and checking.

[0016] In an implementation manner of the present application, in the step S3, the process of traversing user records according to a virtual record pointer, and cross-checking and repairing a traversal path in combination with a page directory slot, is specifically as follows:

[0017] Take the virtual minimum record in the page as a starting point, sequentially traverse user records along next pointers between records, and stop until the virtual maximum record; simultaneously, load the page directory slot at the page tail, each directory slot manages a continuous record interval, and takes the last record in the slot as a management record;

[0018] In the traversal process, the currently parsed record address is periodically compared with the management record address pointed by the directory slot pointer; when the traversal path is interrupted or jumps abnormally due to damage of the record pointer, the management record address of the next valid directory slot is used to correct and reset the current record address to be parsed, so as to repair the traversal path, bypass the local damage point and continue to recover subsequent records.

[0019] In an implementation manner of the present application, in the step S3, the user records are extracted and marked one by one, and specifically:

[0020] When the data content of a single user record is parsed, the parsing is performed according to the field definition; if the parsing process fails due to mismatch of data types or abnormal external storage page, the record is marked as data damage;

[0021] If the parsing is successful, but the data page where the record is located is determined as a damaged page in the first-stage processing flow, the record is marked as possible damage; if the parsing is successful and the data page is complete, the record is marked as no damage; all the recovered records and the corresponding state marks are stored into a result file.

[0022] In an implementation manner of the present application, in the step S3, the row-level data recovery process of the plurality of leaf node pages is executed in multi-thread concurrent mode, and specifically: all the screened leaf node pages are evenly distributed to a plurality of independent working threads, each thread independently executes the row-level recovery flow, and the data results recovered by each thread are summarized.

[0023] In an implementation manner of the present application, the method further comprises:

[0024] The input parameters are received to locate the recovery target; the parameters at least include: a database name to which the data table to be recovered belongs, a table name of the target data table, a disk directory path where the data file is stored, and a configuration file path of a GoldenDB instance;

[0025] The database metadata is automatically loaded according to the parameters, and the corresponding physical data file is located.

[0026] In an implementation manner of the present application, in the step S4, the output row-level data is organized into a structured SQL statement file; each recovered user record generates a corresponding INSERT statement according to the field value, and the mark of data damage or the mark of possible damage is attached to the corresponding statement in the form of SQL comment.

[0027] In an implementation form of the present application, the step S2 of screening the clustered index leaf node page is based on the B+ tree index structure characteristics of the GoldenDB storage engine; only the leaf node page of the clustered index directly contains complete user record row data, and the non-leaf node page only contains index key values for navigation, so the recovery method accurately locates the recovery target to the leaf node page with a level of 0, ensuring that the recovery operation directly acts on the effective data entity.

[0028] In another aspect, the present application also provides a row-level data recovery device for damaged files of a GoldenDB database, characterized in that the device comprises: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to complete the foregoing row-level data recovery method for damaged files of a GoldenDB database.

[0029] The row-level data recovery method and device for damaged files of a GoldenDB database provided by the present application have the following beneficial effects:

[0030] 1. By accurately positioning the leaf node page with a level of 0 of the clustered index in the GoldenDB storage engine, focusing on the effective data entity containing complete user records, breaking through the granularity limitation of traditional page-level recovery, and realizing the extraction and recovery of only row-level data when the data file is damaged, the method avoids the loss of whole page data caused by single page damage, maximizes the retention of effective data, and significantly improves the data recoverability in abnormal scenarios.

[0031] 2. By traversing the user records along the next pointer starting from the virtual minimum record, and periodically cross-checking the management record addresses of the page directory slot, the method corrects and resets the traversal address when the pointer is damaged or the path is abnormal, realizes the bypassing of locally damaged data pages and the continuous recovery of subsequent records, guarantees the continuity of data extraction, and improves the integrity of the recovery result in complex damage scenarios.

[0032] 3. By using a multi-thread concurrent execution mechanism, the selected leaf node page set is evenly distributed to multiple independent working threads, each thread completes row-level recovery and result aggregation in parallel, realizes the efficiency improvement of large-scale data recovery tasks, greatly shortens the recovery time consumption, adapts to the batch data recovery needs of multiple tables or large databases, and enhances the practicality and scene adaptability of the method.

[0033] 4. By organizing the recovered row-level data into a structured INSERT statement, and clearly marking the data "damaged", "possibly damaged", and "undamaged" in the form of SQL comments, intuitive presentation and convenient reuse of the recovery results are achieved, which facilitates users to quickly import the database or carry out manual verification, reduces the subsequent data processing cost, and improves the use reliability and operation convenience after data recovery. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the principles of the application. In the drawings:

[0035] Figure 1 A first-stage flowchart provided for the embodiments of the application;

[0036] Figure 2 A second-stage flowchart provided for the embodiments of the application;

[0037] Figure 3 A GoldenDB database damaged file row-level data recovery device schematic diagram provided for the embodiments of the application. DETAILED DESCRIPTION

[0038] To make the objectives, technical solutions, and advantages of the application clearer, the technical solutions of the application will be described below in conjunction with the embodiments of the application and the corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the application, but not all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the application.

[0039] The embodiments of the application provide a GoldenDB database damaged file row-level data recovery method and device, which mainly includes the following steps:

[0040] Step S1: Scan the GoldenDB data file to be recovered, identify and cache all data pages;

[0041] Step S2: Based on the metadata of the data page, filter out the cluster index leaf node page containing user data from the cached data page;

[0042] Step S3: For each leaf node page filtered out, parse its internal record organization structure, and perform row-level data recovery; the row-level data recovery includes traversing user records according to virtual record pointers, and cross-checking and repairing the traversal path in combination with the page directory slot, so as to extract and mark user records one by one;

[0043] Step S4: output all the recovered row-level data.

[0044] For the convenience of the description of the technical solutions, first introduce the organization mode of data page in GoldenD database storage engine. In GoldenDB database system, the storage engine manages and stores data in units of "pages". The data page, as the smallest I / O read-write unit in the storage engine, is finely divided into multiple functional areas in its internal structure, which are used to store metadata, data records, in-page control information, and other contents. Understanding the organization mode of the data page of GoldenDB database is of key significance to the implementation of fine-grained data recovery. A standard data page (index page) mainly includes the following parts in physical structure:

[0045] 1. File header: located at the start of the page, occupying 38 bytes, used to record the metadata of the data page. The metadata used in the data recovery process includes:

[0046] 1) Page checksum: used for data integrity check, which can be used to judge whether the page is damaged or not complete during recovery;

[0047] 2) Log sequence number (LSN): the last transaction log sequence number of modification, used for crash recovery and consistency check;

[0048] 3) Page type: identifies the type of the page, such as index page, system page, rollback page, etc. Data is only stored in the index page, and only the data in the index page is recovered during data recovery;

[0049] 2. Page header: immediately following the file header, saving the organization information of the in-page records, describing the distribution, state and auxiliary navigation information of the records. The metadata used in data recovery includes:

[0050] 1) Leaf node level: indicates whether the index page is a leaf node. The leaf level is 0, and data is only stored in the leaf node of the clustered index.

[0051] 2) Index ID: identifies the index to which the current page belongs, used to distinguish clustered index and secondary index page.

[0052] 3. Minimum virtual record and maximum virtual large record: as the boundary sentinel of the user record area, auxiliary in-page sorting and searching.

[0053] 4. User record area: stores specific row data, which is logically arranged in primary key or index order. In the B+ tree structure, all actual data is stored in the leaf node page, so the user records of the leaf node page are the core recovery objects in the data recovery process.

[0054] 5. Free space area: used to store subsequent inserted data records, and possibly contain residual data that has been deleted but not yet overwritten.

[0055] 6. Page directory: used to quickly locate records within a page. The directory is stored in reverse order before the end of the file, each directory entry is called a "slot", each slot manages 4-8 user records, and the last record in each slot is the management record of the slot. The directory slots are in reverse order of position and increase, when searching, first do a binary search in the directory, and then traverse the 4-8 records managed by the slot to improve search efficiency.

[0056] 7. File tail: located at the end of the page, occupying 8 bytes, used to check the integrity of the page in both directions, containing the checksum and the last four bits of the log sequence number, which corresponds to the checksum and the last four bits of the log sequence number in the file header.

[0057] The application provides a row-level data recovery method for maximizing the recovery of original data when the data file is damaged. The method includes two stages:

[0058] In the first stage, the leaf nodes of the undamaged cluster index and all damaged pages are scanned and cached to obtain all data pages containing data to be recovered;

[0059] In the second stage, the extracted data pages are subjected to row-level recovery to achieve accurate reconstruction of the original data.

[0060] Specifically, as shown in Figure 1 the execution flow of the first stage is as follows:

[0061] 1. Initialize the cache file: before the recovery starts, create two independent data cache files to cache the following contents: undamaged leaf nodes of the cluster index; all confirmed damaged data pages.

[0062] 2. Load the file to be recovered: open the data file to be recovered, split the data file into multiple data pages according to the data page size in the configuration file, and read the first data page.

[0063] 3. Parse the data page header and tail: read the checksum, log sequence number, page type from the header of the data page, and the checksum, low four bits of the log sequence number in the tail.

[0064] 4. Integrity check: Calculate the checksum based on the content of the data page, and compare it with the checksum in the file header and file tail, and compare the low four bits of the log sequence number in the file tail with the low four bits of the log sequence number in the file header: if the comparison result is consistent, it is determined that the data page has not been damaged; if the comparison result is inconsistent, it is determined that the data page has been damaged, and the data page is added to the cache file of the determined damaged data page, and the analysis work of the next page is started.

[0065] 5. Index page judgment: judge whether the page type is an index page; if it is an index page, continue to the next step of judgment, if it is not an index page, discard the data page, and jump to step 9 to analyze the next page.

[0066] 6. Analyzing page header information: continue to analyze the page header information of the index page, and extract the index number and its level in the index tree.

[0067] 7. Index ID judgment: judge whether the index number is consistent with the target cluster index number, if not, discard the page, otherwise proceed to the next step of judgment.

[0068] 8. Index page level judgment: when the level is 0, it is determined that the page is a leaf node page, and it is added to the file storing the "undamaged cluster index leaf node"; when the level is greater than 0, the page is skipped.

[0069] 9. Determine whether there is a next page, if there is a next page, read the next data page, and jump to step 3 to continue the analysis; otherwise, end the first stage of the process.

[0070] As shown in Figure 2 The second stage carries out row-level recovery on each data page extracted, and cross-checks and repairs using the association relationship between data rows and the organization structure characteristics of the directory slot to maximize the recovery of valid user records. The core idea is to verify each other through the traversal path established by the virtual record and the directory slot pointer, and correct the traversal path at the damaged place through the directory slot. The recovery process is as follows.

[0071] 1. Virtual record analysis: First, analyze the virtual minimum record and the virtual maximum record. The pointer of the virtual minimum record points to the first data of the user, and the user data is connected through the pointer. The pointer of the last record of the user's record points to the virtual maximum record. Therefore, all user data can be traversed through the minimum virtual record and the maximum virtual record.

[0072] 2. Set initial record and directory slot: the record address pointed by the virtual minimum record is the current record to be parsed; the record address pointed by the second directory slot is the management record of the current directory slot, which is the last record address in the current directory slot.

[0073] 3. Check record address range: determine whether the current record pointer address is within the page range. If not, skip the record parsing and directly jump to step 10 to repair the traversal path and continue the recovery.

[0074] 4. Check whether it is a virtual maximum record: determine whether the current record address to be parsed is a virtual maximum record address. If yes, it means that all records in the current page have been parsed, and the parsing process of the current page is terminated.

[0075] 5. Check the number of directory slots: if the number of parsed directory slots exceeds the theoretical maximum value 280, it means that it is impossible to find more recovery records through the directory slots, and the parsing process of the current page is terminated. Otherwise, proceed to the next step.

[0076] 6. Parse and recover the data content pointed by the current record pointer, and mark the recovered data according to the recovery process:

[0077] 1) Parsing exception: if an exception occurs during parsing (such as mismatch between parsed record value and field type, large field external storage page verification failure, or parsing exception, etc.), mark the data with parsing exception as damaged data in the form of comments.

[0078] 2) Normal parsing but page damage: if no exception occurs during parsing, but it is found that the page is damaged during pre-page verification (i.e. from the storage file of "confirmed damaged data pages" in the first stage), mark this data as possibly damaged.

[0079] 3) If no exception occurs during parsing, mark this data as not damaged.

[0080] 7. Store the data and its mark in the result file.

[0081] 8. After the record parsing is completed, add 1 to the number of currently parsed records.

[0082] 9. Directory slot state update: determine whether the current record is a directory slot management record. If yes, move to the next directory slot, and reset the number of records to be parsed in the current directory slot to 0.

[0083] 10. Check the number of records parsed in the directory slot: determine whether the current number of parsed records is greater than or equal to 8. If it is greater than or equal to 8, it means that the next pointer is damaged, and the current record to be parsed is set as the management record of the current directory slot, the parsing path is repaired, and the next directory slot is moved, and the number of parsed records of the current directory slot is reset to -1. Otherwise, the current record to be parsed is set as the next record.

[0084] 11. Jump to step 3 to continue parsing.

[0085] The above process can be executed in multiple threads, and the data pages are evenly distributed to each thread for concurrent parsing to speed up data recovery.

[0086] The above is a row-level data recovery method for GoldenDB database damaged files provided by the embodiment of the present application. Based on the same inventive concept, the embodiment of the present application also provides a row-level data recovery device for GoldenDB database damaged files, Figure 3 A GoldenDB database damaged file row-level data recovery device provided by the embodiment of the present application is shown in FIG. 1, which mainly includes at least one processor 301 and a memory 302 in communication connection with the at least one processor. The memory 302 stores instructions executable by the at least one processor 301, and the instructions are executed by the at least one processor 301 to enable the at least one processor 301 to complete the above-mentioned GoldenDB database damaged file row-level data recovery method. Figure 3

[0087] The following is an example of the present application in a specific scenario: when using the present scheme, the user needs to specify the related parameter information of the database to be recovered in the command line, including:

[0088] The table name to be recovered (–table_name);

[0089] The database name (–database_name);

[0090] The data directory path where the data file to be recovered is stored (–data);

[0091] The address of the corresponding database configuration file (–etc).

[0092] The basic calling mode of the tool is as follows:

[0093] gdb_recovery –table_name=table name –database_name=database name –data=data directory address –etc=configuration file address. ​

[0094] After executing the command, the system loads the metadata configuration of the specified database according to the input parameter information, and automatically locates to the corresponding data file directory. After the recovery is completed, the system outputs the result in a structured form, which can be saved as a SQL file, facilitating subsequent import into a database or manual verification.

[0095] The tool of the embodiment supports parameterized calling and batch task mode, and can perform batch data recovery on multiple tables or multiple database instances in a scripted manner, thereby improving the recovery efficiency in large-scale damage scenarios.

[0096] Each of the embodiments in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment mainly explains the difference from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the related parts can be referred to the part of the method embodiments.

[0097] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or further include elements inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0098] The above only describes the embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various changes and modifications to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of the claims of the present application.

Claims

1. A method for row-level data recovery of a GoldenDB database corrupted file, the method comprising: The method comprises the following steps: ​ Step S1: scanning the GoldenDB data file to be recovered, identifying and caching all data pages; Step S2: based on the metadata of the data pages, filtering out the cluster index leaf node pages containing user data from the cached data pages; the step of filtering out the cluster index leaf node pages is based on the B+ tree index structure characteristics of the GoldenDB storage engine; only the leaf node pages of the cluster index directly contain complete user record row data, and the non-leaf node pages only contain index key values for navigation; Step S3: for each leaf node page filtered out, parsing the internal record organization structure, and performing row-level data recovery; the row-level data recovery comprises traversing user records according to a virtual record pointer, and cross-checking and repairing a traversal path in combination with a page directory slot to extract and mark user records one by one; the process of traversing user records according to a virtual record pointer and cross-checking and repairing a traversal path in combination with a page directory slot is as follows: taking a virtual minimum record in the page as a starting point, sequentially traversing user records along the next pointers between records until a virtual maximum record; at the same time, loading a page directory slot at the end of the page, each directory slot managing a continuous record interval, and taking the last record in the slot as a management record; in the traversal process, periodically comparing the current parsed record address with the management record address pointed to by the directory slot pointer; when the traversal path is interrupted or abnormally jumps due to record pointer damage, the management record address of the next valid directory slot is used to correct and reset the current to-be-parsed record address, thereby repairing the traversal path, bypassing the local damage point and continuing to recover subsequent records; Step S4: outputting all recovered row-level data.

2. The GoldenDB database corrupted file row-level data recovery method of claim 1, wherein, The step S1 further comprises sequentially reading each data page in the data file, and performing integrity checking and attribute analysis on the data page, specifically comprising the following processes: calculating the checksum of the data page and comparing it with the checksum stored in the page, while verifying the consistency of the log sequence numbers at the file head and the file tail, to determine whether the data page is damaged and to separately cache the damaged page; for the data page that passes the integrity check, parsing the page type in the file head of the data page, and filtering out the data page of the index page type; parsing the page header information of the index page to obtain its index ID and level in the B+ tree; caching the leaf node page with the index ID matching the target cluster index ID and the level of 0, to complete the positioning and collection of the effective data bearing page.

3. The GoldenDB database corrupt file row-level data recovery method of claim 2, wherein, The organization structure of the data page comprises a file head at the beginning of the page, a page head for storing record management information in the page, virtual minimum and maximum records as traversal boundaries, a user record area for storing actual row data, a free space area for subsequent insertion, a page directory for quickly locating records, and a file tail at the end of the page for integrity checking; the row-level data recovery mainly acts on the user record area and relies on the page directory and virtual records for navigation and checking.

4. The GoldenDB database corrupted file row-level data recovery method of claim 1, wherein, In the step S3, the user records are extracted and marked one by one, specifically as follows: In the process of analyzing the data content of a single user record, the field definition is used for analysis. If the analysis process fails due to data type mismatch or external storage page exception, the record is marked as data corruption; If the analysis is successful, but the data page where the record is located is determined as a damaged page in the first stage processing flow, the record is marked as possible corruption; If the analysis is successful and the data page is complete, it is marked as no corruption; all recovered records and their corresponding status marks are stored in the result file.

5. The GoldenDB database corrupt file row-level data recovery method of claim 1, wherein, In step S3, the row-level data recovery process of multiple leaf node pages is executed in multi-threading mode. Specifically, all selected leaf node pages are evenly distributed to multiple independent threads, each thread independently executes the row-level recovery process, and the data results recovered by each thread are collected.

6. The GoldenDB database corrupt file row-level data recovery method of claim 1, wherein, The method further comprises: receiving input parameters to locate the recovery target; the parameters at least include: the database name to which the data table to be recovered belongs, the table name of the target data table, the disk directory path where the data file is stored, and the configuration file path of the GoldenDB instance; loading the database metadata automatically according to these parameters and locating the corresponding physical data file.

7. The GoldenDB database corrupt file row-level data recovery method of claim 1, wherein, In step S4, the output row-level data is organized into a structured SQL statement file; each recovered user record generates a corresponding INSERT statement according to its field value, and the mark of data corruption or the mark of possible corruption is attached to the corresponding statement in the form of SQL comment.

8. A GoldenDB database corruption file row-level data recovery apparatus, characterized by, The device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to complete the row-level data recovery method of the GoldenDB database damaged file according to any one of claims 1-7.

Images