Row-level data recovery method and device for damaged file of GoldenDB database

Abstract

The invention discloses a row-level data recovery method and device for a damaged file of a GoldenDB database, and the method comprises the following steps: S1, scanning a to-be-recovered GoldenDB data file, and recognizing and caching all data pages; s2, on the basis of metadata of the data pages, cluster index leaf node pages containing user data are screened out from the cached data pages; s3, for each screened leaf node page, analyzing an internal record organization structure of the leaf node page, and performing row-level data recovery; the row-level data recovery comprises the steps of traversing user records according to a virtual record pointer, and performing cross check and recovery on a traversal path in combination with a page directory slot so as to extract and mark the user records one by one; and S4, outputting all the recovered row-level data.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and device for row-level data recovery of corrupted files in a GoldenDB database. Background Technology

[0002] In GoldenDB databases, the storage engine is widely used for persisting and managing structured data. Its independent tablespace mechanism allows each table's data to be stored in a separate data file, thereby improving data management flexibility and physical isolation. However, in real-world applications, hardware aging, storage media defects, abnormal system shutdowns, unexpected power outages, and even operating system-level anomalies can lead to corruption of GoldenDB database storage files. This corruption may manifest as missing file header information, abnormal page structures, or failed data block verification. Especially faults caused by silent disk media corruption are often undetectable at the file system or database level. Consistency checks only fail when the database accesses or verifies corrupted pages during operation, leading to abnormal instance termination or service interruption.

[0003] Currently, mainstream data recovery tools are limited to page-level recovery. Due to this coarse-grained recovery, once a single data page is corrupted, all rows within that page are lost, making finer-grained row-level repair impossible. Therefore, there is an urgent need for an innovative data recovery mechanism that can overcome the granularity limitations of existing recovery tools, enabling the rapid location and extraction of row-level data even when the GoldenDB data file structure is partially damaged, thereby effectively improving the database's data recoverability in abnormal scenarios. Summary of the Invention

[0004] This application provides a method and device for row-level data recovery of corrupted GoldenDB database files to solve the above-mentioned problems.

[0005] On the one hand, this application provides a row-level data recovery method for corrupted GoldenDB database files, the method comprising the following steps: Step S1: Scan the GoldenDB data file to be recovered, identify and cache all data pages; Step S2: Based on the metadata of the data pages, filter out the clustered index leaf node pages containing user data from the cached data pages; Step S3: For each selected leaf node page, parse its internal record organization structure and perform row-level data recovery; the row-level data recovery includes traversing user records according to the virtual record pointer and performing cross-validation and repair on the traversal path in combination with the page directory slot, so as to extract and mark user records one by one. Step S4: Output all recovered row-level data.

[0006] In one implementation of this application, step S1 further includes: sequentially reading each data page in the data file and performing integrity verification and attribute analysis on it, specifically including the following process: Calculate the checksum of the data page and compare it with the checksum stored in the page. At the same time, verify the consistency of the log sequence number of the file header and file tail to determine whether the data page is corrupted and cache the corrupted page separately. For data pages that pass the integrity check, parse the page type in their file header and filter out data pages of type index page; Parse the header information of the index page to obtain its index ID and level in the B+ tree; Leaf node pages whose index ID matches the target cluster index ID and whose level is 0 are cached, thus completing the location and collection of pages that carry valid data.

[0007] In one implementation of this application, the data page organization structure includes: a file header at the top of the page, a page header for storing record management information within the page, virtual minimum and maximum records serving as traversal boundaries, a user record area for storing actual row data, a free space area for subsequent insertion, a page directory for quickly locating records, and a file tail at the bottom of the page for integrity verification; the row-level data recovery mainly operates on the user record area and relies on the page directory and virtual records for navigation and verification.

[0008] In one implementation of this application, step S3, which involves traversing user records based on virtual record pointers and performing cross-validation and repair on the traversal path using page directory slots, specifically includes: Starting from the smallest virtual record within the page, traverse the user records sequentially along the next pointers between records until the largest virtual record is reached; at the same time, load the page directory slots located at the end of the page, each directory slot manages a continuous record range, and uses the last record in the slot as the managed record; During the traversal, the address of the currently parsed record is periodically compared with the address of the management record pointed to by the directory slot pointer. When the traversal path is interrupted or the jump is abnormal due to the record pointer being damaged, the address of the management record of the next valid directory slot is used to correct and reset the address of the currently parsed record, thereby repairing the traversal path, bypassing the local damage point, and continuing to restore subsequent records.

[0009] In one implementation of this application, step S3, which involves extracting and marking user records one by one, specifically includes: When parsing the data content of a single user record, the parsing is performed according to the field definitions; if the parsing process fails due to data type mismatch or external storage page abnormality, the record is marked as corrupted. If parsing is successful, but the data page containing the record is determined to be a corrupt page in the first stage of the processing flow, then the record is marked as potentially corrupt; if parsing is successful and the data page is intact, then it is marked as not corrupt; all recovered records and their corresponding status flags are stored together in the result file.

[0010] In one implementation of this application, in step S3, the row-level data recovery process for multiple leaf node pages is executed concurrently using multiple threads. Specifically, all the selected leaf node page sets are evenly distributed to multiple independent working threads, each thread independently executes the row-level recovery process, and summarizes the data results recovered by each thread.

[0011] In one implementation of this application, the method further includes: The system receives input parameters to locate the recovery target; the parameters include at least: the database name of the data table to be recovered, the table name of the target data table, the disk directory path where the data file is stored, and the configuration file path of the GoldenDB instance; Based on these parameters, the database metadata is automatically loaded and the corresponding physical data file is located.

[0012] In one implementation of this application, in step S4, the output row-level data is organized into a structured SQL statement file; each recovered user record generates a corresponding INSERT statement based on its field values, and a marker indicating that the data is corrupted or may be corrupted is appended to the corresponding statement in the form of an SQL comment.

[0013] In one implementation of this application, step S2, the step of filtering the leaf node pages of the clustered index, is based on the B+ tree index structure characteristics of the GoldenDB storage engine; only the leaf node pages of the clustered index directly contain complete user record row data, while the non-leaf node pages only contain index key values ​​for navigation. Therefore, the recovery method accurately locates the recovery target to the leaf node page at level 0, ensuring that the recovery operation directly acts on the valid data entity.

[0014] On the other hand, this application also provides a row-level data recovery device for corrupted GoldenDB database files, characterized in that the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to complete the aforementioned row-level data recovery method for corrupted GoldenDB database files.

[0015] This application provides a row-level data recovery method and device for corrupted GoldenDB database files, which has the following advantages: 1. By precisely locating the leaf node page at level 0 of the clustered index in the GoldenDB storage engine, focusing on the valid data entity containing complete user records, it breaks through the granularity limitations of traditional page-level recovery. It enables the extraction and recovery of row-level data only when data files are corrupted, avoiding the loss of the entire page due to single-page corruption, maximizing the preservation of valid data, and significantly improving data recoverability in abnormal scenarios.

[0016] 2. By traversing user records along the next pointer starting from the virtual minimum record, and performing periodic cross-validation using the management record address of the page directory slot, the traversal address is corrected and reset when the pointer is damaged or the path is abnormal. This enables bypassing of partially damaged data pages and continuous recovery of subsequent records, ensuring the continuity of data extraction and improving the integrity of recovery results in complex damage scenarios.

[0017] 3. By adopting a multi-threaded concurrent execution mechanism, the selected leaf node page set is evenly distributed to multiple independent working threads. Each thread completes row-level recovery and result aggregation in parallel, which improves the efficiency of large-scale data recovery tasks, significantly shortens the recovery time, adapts to the batch data recovery needs of multi-table or large databases, and enhances the practicality and scenario adaptability of the method.

[0018] 4. By organizing the recovered row-level data into structured INSERT statements and clearly marking the data status as "corrupted," "potentially corrupted," or "not corrupted" in the form of SQL comments, the recovery results are presented intuitively and reused easily. This allows users to quickly import the data into the database or perform manual verification, reduces subsequent data processing costs, and improves the reliability and ease of use of the recovered data. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1A first-stage flowchart provided for embodiments of this application; Figure 2 A second-stage flowchart provided for embodiments of this application; Figure 3 This is a schematic diagram of a row-level data recovery device for corrupted files in a GoldenDB database, provided as an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] This application provides a method and apparatus for row-level data recovery of corrupted GoldenDB database files. The method mainly includes the following steps: Step S1: Scan the GoldenDB data file to be recovered, identify and cache all data pages; Step S2: Based on the metadata of the data pages, filter out the clustered index leaf node pages containing user data from the cached data pages; Step S3: For each selected leaf node page, parse its internal record organization structure and perform row-level data recovery; the row-level data recovery includes traversing user records according to the virtual record pointer and performing cross-validation and repair on the traversal path in combination with the page directory slot, so as to extract and mark user records one by one. Step S4: Output all recovered row-level data. To facilitate the explanation of this technical solution, we will first introduce the organization of data pages in the GoldenDB database storage engine. In the GoldenDB database system, the storage engine manages and stores data in units of "pages." As the smallest I / O read / write unit in the storage engine, the data page has a finely divided internal structure into multiple functional areas, used to store metadata, data records, page control information, etc. Understanding the organization of data pages in the GoldenDB database is crucial for achieving fine-grained data recovery. A standard data page (index page) mainly includes the following parts in its physical structure: 1. File header: Located at the beginning of the page, occupying 38 bytes, it records the metadata of that data page. The metadata used during data recovery includes: 1) Page checksum: Used for data integrity verification. During recovery, it can be used to determine whether a page is corrupted or intact. 2) Log Sequence Number (LSN): The sequence number of the last modified transaction log, used for crash recovery and consistency verification; 3) Page type: Identifies the page type, such as index page, system page, rollback page, etc. Data is only stored in the index page, and data recovery only restores the data in the index page; 2. Page Header: Following the file header, it stores the organizational information of the records within the page, describing the distribution, status, and auxiliary navigation information of the records. The metadata used in data recovery includes: 1) Leaf node level: Indicates whether the index page is a leaf node. When the leaf level is 0, data is stored only in the leaf nodes of the clustered index.

[0022] 2) Index ID: Identifies the index to which the current page belongs, used to distinguish between clustered indexes and secondary index pages.

[0023] 3. Minimum virtual record and maximum virtual record: These serve as boundary sentinels for the user record area, assisting in in-page sorting and searching.

[0024] 4. User Record Area: This area stores the actual row data, logically arranged according to the primary key or index order. In a B+ tree structure, all actual data is stored in the leaf node pages; therefore, during data recovery, the user records in the leaf node pages are the core recovery objects.

[0025] 5. Free space area: Used to store subsequently inserted data records, and may also contain residual data that has been deleted but not yet overwritten.

[0026] 6. Page Index: Used for quick location of records within a page. The index is located before the end of the file and is stored in reverse order. Each index entry is called a "slot," and each slot manages 4-8 user records. The last record in each slot is the record managed by that slot. The index slots are arranged in reverse order. During a search, a binary search can be performed first in the index, and then the search can be performed on the 4-8 records managed by each slot to improve retrieval efficiency.

[0027] 7. End of file: Located at the end of the page, occupying 8 bytes, it is used for bidirectional verification of page integrity. It contains the checksum and the last four digits of the log sequence number, and its value corresponds to the checksum and the last four digits of the log sequence number in the file header.

[0028] This application provides a row-level data recovery method for maximizing the recovery of original data when data files are corrupted. The method comprises two stages: In the first phase, the leaf nodes of the undamaged clustered index and all damaged pages are scanned and cached to obtain all pages containing data to be recovered. In the second stage, row-level recovery is performed on the extracted data pages to achieve accurate reconstruction of the original data.

[0029] Specifically, such as Figure 1 As shown, the execution flow of the first stage is as follows: 1. Initialize cache files: Before recovery begins, create two separate data cache files to cache the following: leaf nodes of the undamaged clustered index; and all data pages that have been confirmed to be corrupted.

[0030] 2. Load the file to be recovered: Open the data file to be recovered, split the data file into multiple data pages according to the data page size in the configuration file, and read the first data page.

[0031] 3. Parse the data page header and footer: Read the checksum, log sequence number, and page type from the data page header, and the checksum and the lower four bits of the log sequence number from the footer. 4. Integrity Verification: Calculate a checksum based on the content of the data page and compare it with the checksums in the file header and file footer. Simultaneously, compare the lower four bits of the log sequence number in the file footer with the lower four bits of the log sequence number in the file header. If the comparison results match, the data page is determined to be undamaged; if the comparison results do not match, the data page is determined to be damaged, and the data page is added to the cache file of the determined damaged data pages. Proceed to step 9 to start the analysis of the next page.

[0032] 5. Index Page Judgment: Determine if the page type is an index page; if it is an index page, proceed to the next step; if it is not an index page, discard the data page and jump to step 9 to perform analysis on the next page. 6. Parse header information: Continue parsing the header information of the index page to extract the index number and its level in the index tree; 7. Index ID Judgment: Determine whether the index number matches the target cluster index number. If they do not match, discard the page; otherwise, proceed to the next step. 8. Index page level determination: When the level is 0, the page is determined to be a leaf node page and added to the file storing "leaf nodes of undamaged clustered indexes"; when the level is greater than 0, the page is skipped.

[0033] 9. Determine if a next page exists. If a next page exists, read the next data page and jump to step 3 to continue parsing; otherwise, end the first stage of the process.

[0034] like Figure 2As shown, the second stage performs row-level recovery on each extracted data page. It utilizes the relationships between data rows and the organizational structure of the directory slots for cross-validation and repair to maximize the recovery of valid user records. The core idea is to cross-verify the traversal path established through virtual records with the directory slot pointers, and to correct the traversal path at damaged points using the directory slots. The recovery process is as follows.

[0035] 1. Virtual Record Parsing: First, parse the minimum and maximum virtual records. The pointer to the minimum virtual record points to the first record of the user. User records are linked together by pointers. The pointer to the last record of the user points to the maximum virtual record. Therefore, all user data can be traversed using the minimum and maximum virtual records.

[0036] 2. Set initial record and directory slot: The record address pointed to by the virtual minimum record is used as the current record to be parsed; the record address pointed to by the second directory slot is the management record of the current directory slot, which is the address of the last record in the current directory slot.

[0037] 3. Check the record address range: Determine whether the current record pointer address is within the page range. If it is not within the range, skip the parsing of the record and jump directly to step 10 to repair the traversal path and continue the recovery.

[0038] 4. Check if it is the virtual maximum record: Determine if the address of the record to be parsed is the virtual maximum record address. If it is, it means that all records on the current page have been parsed and the parsing process of the current page is terminated.

[0039] 5. Check the number of directory slots: If the number of parsed directory slots exceeds the theoretical maximum of 280, it means that it is impossible to find more recovery records through directory slots, so terminate the parsing process of the current page; otherwise, continue to the next step of judgment.

[0040] 6. Parse and recover the data content pointed to by the current record pointer, and mark the recovered data according to the recovery process: 1) Parsing error: If an error occurs when parsing a field (such as the parsed record value not matching the field type, large field external storage page verification failure, or parsing error), the data with parsing error will be marked as corrupted by commenting.

[0041] 2) Parsing is normal but the page is corrupt: If no abnormality occurs when parsing the field, but the page is found to be corrupt during the pre-page validation (i.e., it comes from the storage file of the "data page confirmed to be corrupt" in the first stage), this data is marked as potentially corrupt.

[0042] 3) If no errors occur when parsing the field, mark the data as uncorrupted.

[0043] 7. Store the data and its tags in the results file.

[0044] 8. After the record parsing is complete, increment the number of records parsed by 1.

[0045] 9. Directory slot status update: Determine if the current record is a directory slot management record. If they are the same, move it to the next directory slot and reset the number of records to be parsed in the current directory slot to 0.

[0046] 10. Check the number of records parsed in the directory slot: Determine if the number of currently parsed records is greater than or equal to 8. If it is greater than or equal to 8, it means that the next pointer is corrupted. In this case, set the current record to be parsed as the management record of the current directory slot, repair the parsing path, move to the next directory slot, and reset the number of parsed records in the current directory slot to -1. Otherwise, set the current record to be parsed as the next record.

[0047] 11. Proceed to step 3 to continue the analysis.

[0048] The above process can be executed using multi-threading, distributing data pages evenly across each thread for concurrent parsing, thus accelerating data recovery.

[0049] The above describes a row-level data recovery method for corrupted GoldenDB database files provided in this application. Based on the same inventive concept, this application also provides a row-level data recovery device for corrupted GoldenDB database files. Figure 3 A schematic diagram of a row-level data recovery device for corrupted GoldenDB database files provided in this application embodiment is shown below. Figure 3 As shown, the device mainly includes: at least one processor 301; and a memory 302 communicatively connected to the at least one processor; wherein the memory 302 stores instructions that can be executed by the at least one processor 301, and the instructions are executed by the at least one processor 301 to enable the at least one processor 301 to complete the aforementioned row-level data recovery method for a corrupted GoldenDB database file.

[0050] Below is an example of this application in a specific scenario: When using this solution, the user needs to specify the relevant parameter information of the database to be restored in the command line, including: The name of the table to be restored (--table_name); The database name (--database_name); The data directory path (--data) where the data files to be recovered are stored. The corresponding database configuration file address (--etc).

[0051] The basic way to call the tool is as follows: gdb_recovery --table_name=table_name--database_name=database_name--data=data_directory_address--etc=configuration_file_address.

[0052] After executing the command, the system loads the metadata configuration of the specified database based on the input parameters and automatically locates the corresponding data file directory. Once the recovery is complete, the system outputs the results in a structured format, which can be saved as an SQL file for later database import or manual verification.

[0053] The tool in this embodiment supports parameterized invocation and batch task mode, and can perform batch data recovery on multiple tables or multiple database instances through scripting, thereby improving the recovery efficiency in large-scale damage scenarios.

[0054] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0055] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0056] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for row-level data recovery of a GoldenDB database corruption file, characterized in that, The method comprises the following steps: Step S1: scanning the GoldenDB data file to be recovered, identifying and caching all data pages; Step S2: based on the metadata of the data pages, filtering out the cluster index leaf node pages containing user data from the cached data pages; Step S3: for each filtered leaf node page, analyzing its internal record organization structure, performing row-level data recovery; the row-level data recovery comprises traversing user records according to virtual record pointers, and cross-checking and repairing the traversal path in combination with the page directory slot, so as to extract and mark user records one by one; Step S4: outputting all recovered row-level data.

2. The GoldenDB database corrupted file row-level data recovery method of claim 1, wherein, The step S1 further comprises sequentially reading each data page in the data file, and performing integrity check and attribute analysis on the data page, specifically comprising the following processes: calculating the checksum of the data page and comparing it with the checksum stored in the page, while verifying the consistency of the log sequence numbers at the file head and the file tail, to determine whether the data page is damaged and to cache the damaged page separately; for the data page that passes the integrity check, analyzing the page type in the file head of the data page, and filtering out the data page of the index page type; analyzing the page head information of the index page to obtain its index ID and level in the B+ tree; caching the leaf node page with the matching index ID and level 0 of the target cluster index ID, to complete the positioning and collection of the valid data bearing page.

3. The GoldenDB database corrupt file row-level data recovery method of claim 2, wherein, The organization structure of the data page comprises: a file head at the beginning of the page, a page head for storing record management information in the page, virtual minimum and maximum records as traversal boundaries, a user record area for storing actual row data, a free space area for subsequent insertion, a page directory for quickly locating records, and a file tail at the end of the page for integrity check; the row-level data recovery mainly acts on the user record area, and relies on the page directory and virtual records for navigation and check.

4. The GoldenDB database corrupt file row-level data recovery method of claim 1, wherein, In the step S3, the process of traversing user records according to virtual record pointers and cross-checking and repairing the traversal path in combination with the page directory slot is specifically: taking the in-page virtual minimum record as the starting point, sequentially traversing the user records along the next pointers between the records until the virtual maximum record; at the same time, loading the page directory slot at the end of the page, each directory slot manages a continuous record interval, and the last record in the slot is taken as the management record; in the traversal process, periodically comparing the currently parsed record address with the management record address pointed to by the directory slot pointer; when the traversal path is interrupted or abnormally jumps due to damaged record pointers, the management record address of the next valid directory slot is used to correct and reset the current to-be-parsed record address, so as to repair the traversal path, bypass the local damage point and continue to recover the subsequent records.

5. The GoldenDB database corrupt file row-level data recovery method of claim 1, wherein, In the step S3, the user records are extracted and marked one by one, specifically: when parsing the data content of a single user record, the fields are parsed according to the field definition; if the parsing process fails due to mismatched data types or abnormal external storage pages, the record is marked as damaged data. If the record is successfully parsed, but the data page where the record is located is determined as a damaged page in the first stage processing flow, the record is marked as possibly damaged; If the record is successfully parsed and the data page is intact, it is marked as not damaged; all recovered records and their corresponding status marks are stored together in the result file.

6. The GoldenDB database corrupt file row-level data recovery method of claim 1, wherein, In the step S3, the row-level data recovery process of the plurality of leaf node pages is executed in multi-thread concurrent mode, specifically, all the screened leaf node pages are evenly distributed to a plurality of independent working threads, each thread independently executes the row-level recovery process, and the data results recovered by each thread are summarized.

7. The GoldenDB database corrupt file row-level data recovery method of claim 1, wherein, The method further comprises: receiving input parameters to locate the recovery target; the parameters at least include: the database name to which the data table to be recovered belongs, the table name of the target data table, the disk directory path where the data file is stored, and the configuration file path of the GoldenDB instance; loading the database metadata automatically according to the parameters and locating the corresponding physical data file.

8. The GoldenDB database corrupt file row-level data recovery method of claim 1, wherein, In the step S4, the output row-level data is organized into a structured SQL statement file; each recovered user record generates a corresponding INSERT statement according to its field value, and the mark of damaged data or the mark of possibly damaged data is attached to the corresponding statement in the form of SQL comment.

9. The GoldenDB database corrupt file row-level data recovery method of claim 1, wherein, In the step S2, the step of screening the clustered index leaf node pages is based on the B+ tree index structure characteristics of the GoldenDB storage engine; only the leaf node pages of the clustered index directly contain complete user record row data, while the non-leaf node pages only contain index key values for navigation, so the recovery method accurately locates the recovery target to the leaf node pages with level 0, ensuring that the recovery operation directly acts on the effective data entity.

10. A GoldenDB database corruption file row-level data recovery apparatus, characterized by, The device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to complete the row-level data recovery method of the GoldenDB database damaged file according to any one of claims 1-9.

Images