Method for restoring progress and status of data collection based on database backup assertions

Abstract

This invention belongs to the field of database recovery technology. It discloses a method for assessing recovery progress and status based on data acquisition and assertion during database backup. The method includes an assertion data acquisition module, an assertion judgment generation module, an assertion judgment module, and an assertion output module. The data acquisition module collects backup data as baseline data for data recovery. The assertion judgment generation module performs pre-assertion judgments based on the baseline data, using these pre-assertion judgments as assertion data for the assertion judgment module during recovery. During database recovery, different assertion judgments are executed at different recovery stages to generate assertion results. These results are then used to determine the status and progress at each stage.

Description

Technical Field

[0001] This invention relates to the field of database recovery technology, and more specifically, to a method for asserting recovery progress and status based on data acquisition during database backup. Background Technology

[0002] Databases are one of the most important assets of an enterprise, and database backup and recovery has become an indispensable routine task for enterprises.

[0003] To ensure that data can be promptly and effectively recovered in the event of deletion or damage due to human error, improper operation, or intentional acts, without significantly impacting business operations, data backup is essential. Data backup is the last line of defense for data disaster recovery. Even with a two-site, three-center architecture, backup remains crucial. If a backup fails, affecting transaction operations or rendering the backup data unrecoverable, these consequences are unbearable for any enterprise. Therefore, the validity of the backup machine is paramount, and verifying the backup machine is essentially performing database recovery operations.

[0004] Database recovery is a lengthy process. Before performing a recovery operation, it's crucial to determine the current recovery status and progress. For example, Chinese patent application CN105528264B discloses a data recovery method and system to prevent accidental operations. The method includes: acquiring all update operations applied to the source database; comparing the update operations with preset rules; if the update operation does not violate the preset rules, executing the update operation in the backup data; if the update operation violates the preset rules, determining whether the update operation is unreasonable; if so, restoring the changes to the source database data caused by the operation; otherwise, executing the update operation in the backup data.

[0005] However, when restoring the database, it cannot display different statuses and progresses for different recovery stages. It can only display the basic data size recovery progress. When different statuses occur, causing recovery failure and the progress cannot proceed, it cannot accurately determine and re-perform the recovery.

[0006] In view of this, the present invention provides a method for asserting recovery progress and status based on data acquisition during database backup. Summary of the Invention

[0007] To overcome the aforementioned deficiencies of the prior art, the present invention provides a method for asserting recovery progress and status based on data acquisition during database backup.

[0008] Including the following steps:

[0009] S1. Obtain the baseline data during database backup;

[0010] S2. Pre-generate different assertion judgments for each stage of recovery based on the baseline data, as assertion data during recovery;

[0011] S3. During recovery, monitor the database and perform assertion judgments based on the assertion data;

[0012] S4. Output the results of assertion judgments synchronously;

[0013] S5. Based on the assertion judgment of the execution result, when restoring backup data, the system can realize the progress and status of the restoration.

[0014] Preferably, in step S1, when the database backup begins, the assertion data acquisition module is used to obtain the data during the database backup, which is then used as the baseline data for data recovery.

[0015] Preferably, step S2 assertion judgment uses an assertion judgment generation module. The benchmark data is input into the assertion judgment generation module, and different assertions for each stage of recovery are pre-generated based on the benchmark data, which serve as assertion data during recovery.

[0016] Preferably, step S3, which performs assertion judgment based on assertion data, uses an assertion judgment module to monitor the database recovery process and execute corresponding assertions based on different assertion data.

[0017] Preferably, the result output in step S4 uses an assertion output module to synchronously output the assertion judgment execution result of the assertion judgment module;

[0018] The data acquisition module collects backup data as the baseline data for data recovery. The assertion generation module makes pre-assertions based on the baseline data and uses these pre-assertions as the assertion data for the assertion module during recovery. During the recovery operation, the database progress and status are monitored based on the results of the assertion module.

[0019] Preferably, the benchmark data acquisition includes: acquiring the average bandwidth of the connection established between the database source and the backup machine;

[0020] Data is collected during the backup process.

[0021] Preferably, the data types in the backup process include:

[0022] Database machine resource basic data, including CPU computing power, I / O devices and memory size;

[0023] The size of the database's metadata;

[0024] The size of the user data in the database;

[0025] Index files, which include the index structure information of the database tables;

[0026] Table structure file, including the definitions of database tables;

[0027] Backup parameters, including the compression algorithm and the number of backup processes.

[0028] Preferably, the method for generating the assertion judgment generation module includes:

[0029] s1, Transmission Phase: Based on the average bandwidth of the connection established between the database source and the backup machine and the size of the backup set, generate the progress of transmission time and size.

[0030] s2, the recovery start phase, asserts whether the recovery is normal based on account permissions and the backup set decompression process;

[0031] S3, during the recovery phase, asserts the recovery progress based on the prepare, apply, and copyBack processes.

[0032] s4, the recovery completion phase, asserts whether the recovery is complete based on the database process, database startup log information, read permissions, and redo log replay process.

[0033] Preferably, the transmission phase assertion includes:

[0034] Status and time progress assertion: If the bandwidth is normal, the status is asserted as "transmitting". The estimated total time is estimated based on the average network bandwidth and backup set size. The unit time in "transmitting" is the time already transmitted. The time already transmitted / total time is the time progress.

[0035] Status and transmission progress assertions are based on the backup set size as the total size. The assertion of transmitted size / total size is the transmission progress, and the transmitted size is an increasing status assertion indicating that transmission is in progress.

[0036] Preferably, the recovery initiation phase includes:

[0037] Account permission assertion: if umark equals 755 (True), proceed to the next assertion; otherwise, assertion recovery fails.

[0038] The backup set decompression process asserts that the backup set exists, checks the file size at the corresponding path, calculates the time taken based on CPU usage, and if True, the status is "recovery started"; otherwise, it enters the recovery assertion stage.

[0039] Preferably, the recovery phase includes:

[0040] The prepare process asserts that the prepare process exists. If True, the status is in recovery; if False, the recovery assertion is entered.

[0041] The apply process assertion is based on the apply file size. If the apply file size is equal to the source file size, the status is "apply in progress," and the progress is the size restored per unit time / the source file size.

[0042] If the copyBack process is asserted to exist, its size is checked, its status is in the copyBack stage, and its progress is the size copied / total size.

[0043] Preferably, the recovery termination phase includes;

[0044] The database process is asserted to exist and its status is Running. If True, the process proceeds to the next assertion; otherwise, an internal check is performed.

[0045] Database startup log information, keyword search assertion, if True, proceed to the next assertion, if False, the status is startup in progress;

[0046] Readable, asserts that the select was successful, True proceeds to the next assertion, False executes an internal check;

[0047] The redo log replay process is asserted to exist. If True, the status is in replay mode; otherwise, proceed to the next assertion.

[0048] The technical effects and advantages of this invention, which uses data acquisition assertions to monitor recovery progress and status during database backup, are as follows:

[0049] 1. This invention collects data on corresponding indicators during database backup as a pre-generated recovery assertion judgment data.

[0050] 2. During recovery, based on pre-generated assertion judgments, different assertion judgments are executed at different recovery stages when the database is being recovered, and assertion results are generated. The status and progress of different stages are then determined in combination with the assertion results. Attached Figure Description

[0051] Figure 1 This is a flowchart of the data acquisition process during database backup in this invention;

[0052] Figure 2 The flowchart of the recovery assertion module and assertion output module of this invention is shown below;

[0053] Figure 3 This is an assertion logic diagram of the assertion generation module of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] This invention provides, for example Figure 1-3 The method shown, which asserts recovery progress and status based on data acquisition during database backup, includes the following steps:

[0056] S1. Obtain the baseline data during database backup;

[0057] S2. Pre-generate different assertion judgments for each stage of recovery based on the baseline data, as assertion data during recovery;

[0058] S3. During recovery, monitor the database and perform assertion judgments based on assertion data;

[0059] S4. Synchronously output the execution results of the assertion judgment module;

[0060] S5. Based on the assertion judgment of the execution result, when restoring backup data, the system can realize the progress and status of the restoration.

[0061] Specifically, in step S1, when the database backup begins, the assertion data acquisition module is used to obtain the data during the database backup process, which serves as the baseline data for data recovery. The baseline data includes: the average bandwidth of the connection established between the database source and the backup machine, and the data during the backup process. Establishing a connection refers to establishing a communication channel or network connection between the database source and the backup machine so that they can transfer data to each other and perform backup operations.

[0062] More specifically, ensure that both the backup machine and the database source are correctly set up and configured, and that the network connection is normal. Determine the database type to be collected (e.g., MySQL, Oracle, SQL Server) and the corresponding connection method (e.g., ODBC, JDBC). Install the necessary database clients or drivers on the backup machine to enable communication with the database source. These clients or drivers can be downloaded and installed according to the database type and vendor. Based on the database type and connection method, use appropriate connection strings or configuration information to create a database connection on the backup machine, specifying the hostname or IP address, port number, database name, username, and password of the database source. After successful connection, use an ETL (Extract, Transform, Load) tool to perform data extraction. Extract data from the database source using the aforementioned tool and load it into the target location or target database on the backup machine. Determine the collection time for bandwidth data collection and the amount of data collected within the determined collection time. Record the amount of data transferred from the database source to the backup machine in each time period. In this embodiment, network traffic monitoring tools and database logs are used to obtain this information. The data transmission volume is recorded every 5 minutes to ensure the accuracy of bandwidth speed prediction.

[0063] By collecting the bandwidth information and processing the data, the average bandwidth can be obtained. The amount of data collected in each time period is converted into the number of bytes. The bandwidth of each measurement is calculated. The amount of data collected in each measurement is divided by the measurement time interval to obtain the average data transmission rate per second (in bytes). The bandwidth values ​​of all measurement periods are summarized to calculate the average bandwidth. The bandwidth values ​​of all measurement periods are added together and divided by the total number of measurements to calculate the average bandwidth.

[0064] Furthermore, the data types involved in the backup process include:

[0065] Database machine resource basic data, CPU computing power, CPU (Central Processing Unit) is one of the core components of a computer system, responsible for performing calculation and processing tasks. CPU computing power is usually measured by the processor's clock speed (GHz) or number of cores. Higher clock speed and number of cores usually mean higher computing power and processing speed. Backup time can be estimated based on CPU computing power.

[0066] I / O devices are used to interact with the computer system. In this embodiment, they include hard disk drives and network adapters. The performance of I / O devices affects the data read and write speeds, as well as the system's communication capabilities with external devices.

[0067] Memory size is an important parameter in a computer system, representing the capacity of the system's available random access memory (ROM). In this embodiment, the backup and recovery capacity is assessed by obtaining the ROM capacity.

[0068] The size of the database's metadata ensures that the database's structural integrity is preserved during data recovery, so that the database objects can be correctly reconstructed;

[0069] The size of the user data in the database is used to assess the required storage space and plan the capacity of the storage devices;

[0070] The index file contains the index structure information of the database table. It is automatically created and maintained by the database system to support the indexing of the table. The index can be based on one or more columns and provide fast data lookup, sorting and filtering functions. In this embodiment, the index types include B-tree index, hash index and full-text index. When the index file data is obtained, assertions can be made accordingly during recovery.

[0071] A table structure file contains information such as table name, column names, data types, constraints, and indexes, defining the table structure to retrieve information during database startup and queries. This file is automatically created and maintained by the database system and describes the tables within the database. This example includes a table structure file. MySQL uses storage engines to manage table data and structure. For each storage engine, MySQL creates a corresponding table structure file. PostgreSQL uses tablespaces to manage table data and structure. Oracle databases also use tablespaces to manage table data and structure.

[0072] Backup parameters, compression algorithms during backup, and the number of backup processes. This embodiment uses lossless compression algorithms, including Huffman coding: encoding characters based on their frequency of occurrence, using shorter codes for frequently occurring characters and longer codes for less frequently occurring characters; Lempel-Ziv-Welch (LZW): compressing data by constructing a dictionary and using existing strings in the dictionary to replace repeated strings; Deflate: combining Huffman coding and the LZ77 algorithm, commonly used for compressing ZIP file formats and PNG image formats; DEFLATE: commonly used for data compression in GZIP file formats and the HTTP protocol; obtaining backup process data is an important data protection strategy used to create copies of data so that data can be recovered in the event of data loss, corruption, or disaster. The backup process in this embodiment includes selecting a backup strategy: determining the backup... The backup process involves several steps: 1. **Frequency and Level of Backups:** This includes selecting the backup target location, such as a full backup, incremental backup, or differential backup. A full backup copies the entire dataset, an incremental backup only backs up changes since the last full backup, and a differential backup only backs up changes since the last full or incremental backup. 2. **Backup Execution:** This involves selecting the backup target location, determining where the backup data will be stored (e.g., local disk, network storage, tape library). 3. **Backup Validation:** This involves performing the actual backup operation according to the backup strategy. This may involve copying data to the backup target location, processing the data using compression and encryption techniques, and logging detailed backup operation information. 4. **Backup Verification:** This involves verifying the backup data to ensure its integrity and recoverability. This may include checking the checksum of the backup data, restoring partial data for verification, or performing integrity checks and testing the recovery process. 5. **Logging and Reporting:** This involves logging backup operation information, including backup time, backup data size, and whether the backup was successful.

[0073] Step S2 assertion judgment uses the assertion judgment generation module. The benchmark data is input into the assertion judgment generation module, and different assertions for each stage of recovery are pre-generated based on the benchmark data, which are used as assertion data during recovery.

[0074] Step S3: Perform assertion judgment based on assertion data. The assertion judgment module monitors the database recovery process and executes corresponding assertions based on different assertion data.

[0075] Step S4 output uses the assertion output module to synchronously output the assertion judgment execution results of the assertion judgment module;

[0076] The data acquisition module collects backup data as the baseline data for data recovery. The assertion generation module makes pre-assertions based on the baseline data and uses these pre-assertions as the assertion data for the assertion module during recovery. During the recovery operation, the database progress and status are monitored based on the results of the assertion module.

[0077] The generation method of the assertion judgment generation module includes:

[0078] s1, Transmission Phase: Based on the average bandwidth of the connection established between the database source and the backup machine and the size of the backup set, generate the progress of transmission time and size.

[0079] s2, the recovery start phase, asserts whether the recovery is normal based on account permissions and the backup set decompression process;

[0080] S3, during the recovery phase, asserts the recovery progress based on the prepare, apply, and copyBack processes.

[0081] s4, the recovery completion phase, asserts whether the recovery is complete based on the database process, database startup log information, read permissions, and redo log replay process.

[0082] Transmission phase assertions include:

[0083] Status and time progress assertion: If the bandwidth is normal, the status is asserted as "transmitting". The estimated total time is estimated based on the average network bandwidth and backup set size. The unit time in "transmitting" is the time already transmitted. The time already transmitted / total time is the time progress.

[0084] The status and transfer progress assertions are based on the backup set size as the total size. The assertion that the transferred size / total size represents the transfer progress is used, with the transferred size incrementing as the status assertion indicating transfer is in progress. For file transfers, the transfer progress can be calculated based on the ratio of the number of bytes transferred to the total number of bytes in the file. For network transfers, the transfer progress can be calculated based on the ratio of the number of data packets transferred to the total number of data packets. For data backups, the transfer progress can be calculated based on the ratio of the amount of data backed up to the total amount of data. The transfer progress display can be real-time updated; that is, the progress bar or percentage updates in real time as the transfer progresses. Additionally, an estimated remaining time can be displayed so that users know the expected completion time of the transfer operation. The transfer progress display serves as a feedback mechanism for users, increasing the visibility and transparency of the operation and helping users understand its progress. When designing and implementing the transfer function, the display of transfer progress was considered to improve the user experience and provide better operational feedback.

[0085] The recovery phase includes:

[0086] Account permission assertion: When umark equals 755, the assertion is True, indicating that the current account has read and write permissions, and then proceeds to the next assertion; when the assertion is False, umark is not equal to 755, and the assertion recovery fails.

[0087] The backup set decompression process, if asserted as True, indicates that the backup machine decompression exists, and the status is recovery started;

[0088] Check the file size at the corresponding path, calculate the time taken based on the CPU. The CPU calculates the time taken based on the memory read / write rate and average bandwidth. If the assertion is True, it means the recovery has started; if the assertion is False, it means the recovery is interrupted.

[0089] The recovery phase includes:

[0090] If the prepare process asserts True, it means the prepare process exists and the status is in recovery. If the assertion is False, it means the prepare process does not exist and the process enters the recovery assertion stage.

[0091] The apply process assertion is based on the apply file size. If the apply file size is equal to the source file size, the status is "apply in progress". The progress is the recovery size per unit time / source file size. The recovery size per unit time is estimated based on CPU computing power, memory read and write speed and average bandwidth.

[0092] If the copyBack process is asserted to exist, its size is checked, and its status is in the copyBack stage. The progress is the copied size / total size. The copied size is estimated based on CPU computing power, memory read / write speed, and average bandwidth.

[0093] The recovery phase includes:

[0094] The database process is asserted to exist and its status is Running. If True, the process proceeds to the next assertion; otherwise, an internal check is performed.

[0095] Database startup log information, keyword search assertion, if True, proceed to the next assertion, if False, the status is startup in progress;

[0096] Readable, asserts select(1) successfully, True proceeds to the next assertion, False executes internal check;

[0097] The redo log replay process is asserted to exist. If True, the status is in replay mode; otherwise, proceed to the next assertion.

[0098] Based on the key data collection during backup, assertion judgments are combined during recovery to determine the recovery status and progress of the database. The recovery phase is refined into various stages: transmission phase, recovery start phase, recovery in progress phase, and recovery end phase. Different assertion judgment indicators are executed based on each stage to determine the specific progress and status of that stage.

[0099] By combining backup set data, machine resources, and network dimensions, and applying various indicator data to different stages of recovery, a more accurate recovery status judgment is achieved. More explicit indicators are provided as a reference for the recovery stage. Data collection of corresponding indicators during backup is used as pre-generated recovery assertion judgment data. During recovery, based on the pre-generated assertion judgment, different assertion judgments are executed at different recovery stages when the database is being restored, generating assertion results. The status and progress of different stages are then judged based on the assertion results.

[0100] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0101] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for asserting the progress and status of a recovery based on data collection at the time of a database backup, characterized by, Including the following steps: S1. Obtain the baseline data during database backup; Step S1: When the database backup begins, use the assertion data acquisition module to obtain the data during database backup and use it as the baseline data for recovery. The acquisition of the benchmark data includes: acquiring the average bandwidth of the connection established between the database source and the backup machine; and acquiring data during the backup process. S2. Pre-generate different assertion judgments for each stage of recovery based on the baseline data, as assertion data during recovery; Step S2 assertion judgment uses the assertion judgment generation module, inputs the baseline data into the assertion judgment generation module, and pre-generates different assertion judgments for each stage of recovery based on the baseline data, as assertion data during recovery. The generation method of the assertion judgment generation module includes: s1, Transmission Phase: Based on the average bandwidth of the connection established between the database source and the backup machine and the size of the backup set, generate the progress of transmission time and size. s2, the recovery start phase, asserts whether the recovery is normal based on account permissions and the backup set decompression process; S3, during the recovery phase, asserts the recovery progress based on the prepare, apply, and copyBack processes. s4, Recovery Completion Phase: Assert whether recovery is complete based on database process, database startup log information, read permissions, and redo log replay process. Transmission phase assertions include: Status and time progress assertion: If the bandwidth is normal, the status is asserted as "transmitting". The estimated total time is estimated based on the average network bandwidth and backup set size. The unit time in "transmitting" is the time already transmitted. The time already transmitted / total time is the time progress. Status and transmission progress assertions: based on the backup set size as the total size, assert the transmitted size / total size as the transmission progress, and the transmitted size as an increasing status assertion indicating that transmission is in progress. The recovery phase includes: Account permission assertion: if umark equals 755 (True), proceed to the next assertion; otherwise, assertion recovery fails. The backup set decompression process asserts that the backup set exists, checks the file size at the corresponding path, calculates the time taken based on CPU, and if True, the status is recovery start; if False, the recovery assertion is entered. The recovery phase includes: The prepare process asserts that the prepare process exists. If True, the status is in recovery; if False, the recovery assertion is entered. The apply process assertion is based on the apply file size. If the apply file size is equal to the source file size, the status is "apply in progress," and the progress is the size restored per unit time / the source file size. If the copyBack process is asserted to exist, then the size is checked, the status is in the copyBack stage, and the progress is the size copied / the total size. The recovery phase includes: The database process is asserted to exist and its status is Running. If True, the process proceeds to the next assertion; otherwise, an internal check is performed. Database startup log information, keyword search assertion, if True, proceed to the next assertion, if False, the status is startup in progress; Readable, asserts that the select was successful, True proceeds to the next assertion, False executes an internal check; The redo log replay process is asserted to exist. If True, the status is in replay; otherwise, proceed to the next assertion. S3. During recovery, monitor the database and perform assertion judgments based on the assertion data; S4. Output the results of assertion judgments synchronously; S5. Based on the assertion judgment of the execution result, when restoring backup data, the system can realize the progress and status of the restoration.

2. The method for assessing recovery progress and status based on data acquisition assertions during database backup, as described in claim 1, is characterized in that... Step S3: Perform assertion judgment based on assertion data. The assertion judgment module monitors the database recovery process and executes corresponding assertions based on different assertion data.

3. The method for assessing recovery progress and status based on data acquisition assertions during database backup, as described in claim 2, is characterized in that... Step S4 output uses the assertion output module to synchronously output the assertion judgment execution results of the assertion judgment module; The data acquisition module collects backup data as the baseline data for data recovery. The assertion generation module makes pre-assertions based on the baseline data and uses these pre-assertions as the assertion data for the assertion module during recovery. During the recovery operation, the database progress and status are monitored based on the results of the assertion module.

4. The method for assessing recovery progress and status based on data acquisition assertions during database backup, as described in claim 1, is characterized in that... The data types included in the backup process are: Database machine resource basic data, including CPU computing power, I / O devices and memory size; The size of the database's metadata; The size of the user data in the database; Index files, which include the index structure information of the database tables; Table structure file, including the definitions of database tables; Backup parameters, including the compression algorithm and the number of backup processes.

Images