Database clone compression backup and recovery method and system

Abstract

This invention relates to the field of database technology and discloses a database cloning, compression, backup, and recovery method and system. By triggering online cloning on database instance nodes during normal database operation and transmitting the cloned data in real-time to the backup request node for compression and disk persistence during the generation process, the backup process can achieve online compressed backup without relying on complex external deployments. This reduces disk space requirements during backup and improves the flexibility of backup deployment. Furthermore, by organizing data page changes into independent data page files and replacing them according to index page and data page sequence numbers during recovery, the compressed backup data can quickly form a file format recognizable by the database instance after decompression. Combined with the database's own recovery mechanism, eventual consistency recovery is achieved, thereby reducing the complexity of backup data recovery and improving the feasibility of the recovery process.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a database cloning, compression, backup, and recovery method and system. Background Technology

[0002] Database instances manage large amounts of user data, but during database operation, various reasons such as database system failures, database instance node machine failures, and human error may cause database anomalies or even prevent normal startup, affecting data security. Therefore, regularly backing up the database system is a reliable and necessary means of preventing failure risks.

[0003] Online data backups are generally required during database instance operation to minimize the impact of the backup operation on business. However, the large volume of backup data also places demands on storage space. Existing database systems typically support online backups, but compression is often difficult to perform simultaneously during the backup process. External tools are often needed to compress the backup data after the backup is complete, which not only complicates operation and maintenance but may also result in the original and compressed backup data existing simultaneously, further increasing storage space requirements. While some external operation and maintenance tools support compressed backups, they introduce additional deployment and dependencies, reducing the flexibility of backup deployment.

[0004] In addition, online backups typically require incremental logs or data page-level change capture to ensure consistency. However, with the introduction of compression, the data content and length change, which makes the recovery process, which is indexed by page number, more complex. A clear correspondence needs to be established between the organization of compressed backup data and the file organization method that the database instance can recognize in order to support the feasibility of subsequent recovery operations.

[0005] Therefore, how to achieve online cloning and online compressed backup during normal database operation, and how to quickly generate a data file recognizable by the database instance based on the compressed backup data during recovery, thereby completing the database instance recovery, remains a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] This invention provides a database cloning, compression, backup, and recovery method and system, aiming to solve at least one of the above-mentioned technical problems.

[0007] To achieve the above objectives, the present invention provides a database cloning, compression, backup, and recovery method, comprising the following steps: Obtain the user's clone compression backup request statement, and parse it to obtain the backup mode, database instance node connection information, compressed backup data storage directory, and compression algorithm configuration parameters; Based on the backup mode, a compressed clone request command is sent to the database instance node to trigger the database instance node to start the cloning task during normal database operation. The database instance nodes are controlled to sequentially generate and output clone data in the cloning task; wherein, the clone data includes at least tablespace data, data page data that has undergone data changes, and redo log data corresponding to the data page data; The cloned data is received, and the tablespace data, data page data, and redo log data are compressed based on the compression algorithm configuration parameters. The compressed data is then written to the compressed backup data storage directory to form compressed backup data. The compressed backup data is copied to the target node to be restored, and the compressed backup data is decompressed on the target node using a compression algorithm that is consistent with the compression algorithm configuration parameters to obtain decompressed backup data. Perform file recovery processing on the decompressed backup data to restore the decompressed backup data to a file format recognizable by the database instance, and start the database instance based on the data file after the file recovery processing is completed to complete the recovery.

[0008] Optionally, based on the backup mode, a compressed clone request command is sent to the database instance node to trigger the database instance node to start the cloning task during normal database operation, specifically including: Based on the backup mode obtained by parsing the user's clone compression backup request statement, determine whether the backup mode is a local clone compression mode or a remote clone compression mode. If the local clone compression mode is determined, the backup request object node and the database instance node are determined to be the same node, and the pre-set compression variables and compression algorithm variables are read to generate the compression algorithm configuration parameters; When the remote cloning compression mode is determined, the database instance node connection information is extracted from the cloning compression backup request statement and a network communication connection is created. The pre-set compression variables and compression algorithm variables are read to generate the compression algorithm configuration parameters. Based on the backup mode and the compression algorithm configuration parameters, a compressed cloning request instruction that can be recognized by the database instance node is constructed.

[0009] Optionally, controlling the database instance nodes to sequentially generate and output clone data in the cloning task specifically includes: Read the database system's metadata dictionary data to obtain a list of tablespace files; During the tablespace copying phase, the tablespace file list is traversed to read the tablespace data sequentially and output it to form the first phase of cloned data; During the data page copying phase, a list of data page numbers that have undergone data changes is determined based on the data change records generated during the tablespace copying phase. The corresponding data page data is then read and output to form the cloned data in the second phase. During the redo log copying phase, the archived redo log data is read and output to form the third-stage cloned data.

[0010] Optionally, during the data page copying phase, a list of data page numbers that have undergone data changes is determined based on the data change records generated during the tablespace copying phase. The corresponding data page data is then read and output, specifically including: Before the tablespace copy phase begins, start the data page archiving record to record the sequence number of the data page that has undergone data changes during the tablespace copy phase; When entering the data page copying stage, the data page archive records are read to form the data page sequence number list, and the data page sequence number list is output all at once; After outputting the data page number list, the index positions of each data page number in the tablespace file are read sequentially, and the data page data at the index positions is read from the tablespace file and output, until the data page data corresponding to the data page number list is output.

[0011] Optionally, the cloned data is received, and the tablespace data, data page data, and redo log data are compressed based on the compression algorithm configuration parameters. The compressed data is then written to the compressed backup data storage directory to form compressed backup data. Specifically, this includes: During the tablespace copying phase, a tablespace compressed file associated with the tablespace name is created for each tablespace file, and the received tablespace data is compressed and written to the tablespace compressed file. Subsequent received data for the same tablespace is appended. During the data page copying phase, a data page file associated with the table space name is created for each table space. The received data page data is encapsulated and compressed according to the preset data page file organization format and written to the data page file. Subsequent received data for the same table space is appended to the data page file. During the redo log copying phase, the received redo log data is compressed and written to the redo log compressed file.

[0012] Optionally, the data page file organization format includes: data blocks and data pages, with the first data page of each data block being an index page; after encapsulation according to a preset data page file organization format, the data page is compressed and written into the data page file, specifically including: A data page file is divided into one or more data blocks, each data block contains several data pages, and the first data page is the index page; Write an identifier field into the first field of the index page to identify whether the current data block is the last data block of the data page file; The remaining fields of the index page are written with the data page number value in sequence, and the data page content corresponding to the data page number recorded in the second field of the index page is stored in the second data page of the data block, thereby establishing the correspondence between the data page number and the data page content one by one. After a data block is full, a new data block is created and appended to the previous data block, and the identifier field is updated when it is determined to be the last data block.

[0013] Optionally, the database cloning, compression, backup, and recovery method further includes: During the tablespace copying phase, multiple clone threads are started to read data output from different tablespaces in parallel. On the backup request object node, start multiple compression threads corresponding to the number of clone threads, so that each compression thread receives the data output by the corresponding clone thread and performs compression writing. During the data page copying phase, multiple clone threads are started to read and output the data of the data pages associated with different tablespaces that have undergone data changes in parallel, while the corresponding compression thread performs compression writing on the output data.

[0014] Optionally, the compressed backup data is copied to the target node to be restored, and decompression processing is performed on the target node using a compression algorithm consistent with the compression algorithm configuration parameters to obtain decompressed backup data, specifically including: The compressed backup data is copied to the target node to be restored, and the compression algorithm configuration parameters are provided as decompression parameters to the data decompression tool. The data decompression tool is controlled to traverse the compressed backup data directory to obtain all compressed files, and each compressed file is decompressed to generate the corresponding decompressed file; Delete each compressed file after it has been decompressed, until the entire compressed backup data directory has been decompressed.

[0015] Optionally, file recovery processing is performed on the decompressed backup data to restore it to a file format recognizable by the database instance, and the database instance is started based on the data file after file recovery processing to complete the recovery. Specifically, this includes: Read the data page file corresponding to the tablespace, and read the index page of each data block to obtain the identification field and data page number; Based on the data page sequence number, obtain the index position in the corresponding tablespace file, and read the data page content corresponding to the data page sequence number in the data page file; The data page content is used to replace the data page data at the index position in the tablespace file to complete the recovery of the corresponding data page, and this process is repeated until the last data block has been processed is determined according to the identifier field. After replacing all the data pages corresponding to the data page file, delete the data page file. The database instance is started based on the tablespace file after the data pages have been replaced, and the data recovery is completed using the database's own recovery mechanism.

[0016] Furthermore, to achieve the above objectives, the present invention also provides a database cloning, compression, backup, and recovery system, comprising: The parsing module is used to obtain the user's clone compressed backup request statement and parse it to obtain the backup mode, database instance node connection information, compressed backup data storage directory, and compression algorithm configuration parameters. The sending module is used to send a compressed clone request instruction to the database instance node based on the backup mode, so as to trigger the database instance node to start the cloning task during normal database operation; The output module is used to control the database instance nodes to sequentially generate and output clone data in the cloning task; wherein, the clone data includes at least tablespace data, data page data that has undergone data changes, and redo log data corresponding to the data page data; The compression module is used to receive the cloned data, and perform compression processing on the tablespace data, the data page data and the redo log data respectively based on the compression algorithm configuration parameters, and write the compressed data into the compressed backup data storage directory to form compressed backup data; The decompression module is used to copy the compressed backup data to the target node to be restored, and to perform decompression processing on the compressed backup data on the target node using a compression algorithm consistent with the compression algorithm configuration parameters to obtain decompressed backup data. The recovery module is used to perform file recovery processing on the decompressed backup data to restore the decompressed backup data to a file format that the database instance can recognize, and to start the database instance based on the data file after the file recovery processing is completed to complete the recovery.

[0017] The beneficial effects of this invention are as follows: It proposes a database cloning, compression, backup, and recovery method and system. By triggering online cloning on database instance nodes during normal database operation and transmitting the cloned data in real time to the backup request node for compression and disk persistence during the generation process, the backup process can achieve online compressed backup without relying on complex external deployments, thereby reducing the disk space requirements during backup and improving the flexibility of backup deployment. Furthermore, by organizing data page change data into independent data page files and replacing them according to the index page and data page sequence number during recovery, the compressed backup data can quickly form a file format recognizable by the database instance after decompression. Combined with the database's own recovery mechanism, eventual consistency recovery is achieved, thereby reducing the complexity of backup data recovery and improving the feasibility of the recovery process. Attached Figure Description

[0018] Figure 1 This is a schematic flowchart of the database cloning, compression, backup, and recovery method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the data page file organization format in an embodiment of the present invention; Figure 3 This is a schematic diagram of the database cloning, compression, backup, and recovery system according to an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] For ease of description, the following participating nodes and data objects are introduced in this embodiment: the backup request object node is used to receive user clone compressed backup request statements and undertake compression and disk writing; the database instance node is used to execute cloning tasks and output cloned data during normal database operation; the target node is used to carry out the recovery process and output a bootable database instance file. The cloned data includes at least tablespace data, data page data that has undergone data changes, and redo log data corresponding to the data page data; the compression algorithm configuration parameters are used to simultaneously constrain compression and decompression processing; the compressed backup data storage directory is used to store compressed backup data and serves as the input directory for subsequent copying and decompression. The above node names are only used for functional differentiation and are not limited to physical single machines, virtual machines, or container environments. This embodiment of the invention is also applicable to multi-node cluster deployments or same-machine deployments.

[0021] This invention provides a database cloning, compression, backup, and recovery method, referring to... Figure 1 , Figure 1This is a schematic flowchart of the database cloning, compression, backup, and recovery method according to an embodiment of the present invention.

[0022] In this embodiment, a database clone compression backup and recovery method includes the following steps: Step S10: Obtain the user's clone compression backup request statement, and parse it to obtain the backup mode, database instance node connection information, compressed backup data storage directory, and compression algorithm configuration parameters.

[0023] In this embodiment of the invention, step S10 is used to form a unified input for the subsequent cloning, compression, decompression and recovery processes before the backup begins, so that each stage can coordinate around the same set of connection objects and storage path algorithm parameters, avoiding decompression failure or unusable recovery results due to inconsistent parameters between the backup side and the recovery side.

[0024] It should be noted that the backup mode parsed in this step is used to determine the method of command sending and data transmission channel establishment; the database instance node connection information is used to establish network communication connection in remote mode; the compressed backup data storage directory is used to hold the compressed file to disk and serve as the source directory for subsequent copying; the compression algorithm configuration parameters are used to limit the compression algorithm type and its optional parameters, and are reused as consistent decompression parameters in the decompression stage.

[0025] Step S20: Based on the backup mode, send a compressed clone request command to the database instance node to trigger the database instance node to start the cloning task during normal database operation.

[0026] In this embodiment of the invention, step S20 is used to initiate the online cloning process without stopping database operations. The compressed cloning request instruction can be regarded as a comprehensive control information that triggers the cloning task, constrains the output content / order, and specifies the backup side's receiving method: on the one hand, it triggers the database instance node to enter the cloning task state; on the other hand, it constrains the cloning task to output tablespace data, data page data, and redo log data in stages during the output process, so that the backup request object node can compress and write the data by type and form a recoverable data set. The compressed cloning request instruction can be implemented by same-machine invocation in local mode, and can be sent on an established network communication connection in remote mode, neither of which constitutes a limitation on the implementation of the present invention.

[0027] In an alternative implementation, step S20 includes the following sub-steps: S201: Based on the backup mode obtained from parsing the user's clone compression backup request statement, determine whether the backup mode is a local clone compression mode or a remote clone compression mode; S202: If it is determined to be a local clone compression mode, determine that the backup request object node and the database instance node are the same node, and read the pre-set compression variables and compression algorithm variables to generate the compression algorithm configuration parameters; S203: If it is determined to be a remote clone compression mode, extract the database instance node connection information from the clone compression backup request statement and create a network communication connection, and read the pre-set compression variables and compression algorithm variables to generate the compression algorithm configuration parameters; S204: Based on the backup mode and the compression algorithm configuration parameters, construct a compressed clone request instruction that can be recognized by the database instance node.

[0028] Specifically, step S201 is used to clarify the subsequent instruction triggering path and data transmission path. In local cloning compression mode, cloning and data transfer can be triggered via in-process signals or shared memory. In remote cloning compression mode, a network channel needs to be established based on node connection information, and instructions and cloning data need to be transmitted within the channel, allowing the backup request node to receive the cloning data stream from the database instance node. Steps S202 and S203 are used to form compression algorithm configuration parameters. These parameters are used for both compression and disk persistence on the backup side and decompression and recovery on the target node, ensuring consistency between compression and decompression. Compression variables and compression algorithm variables can be understood as system-preset or administrator-configured algorithm switches, compression levels, window sizes, parallel parameters, etc., without being limited to a specific algorithm. Step S204 is used to construct a compression cloning request instruction recognizable by the database instance node based on the backup mode and the compression algorithm configuration parameters.

[0029] Step S30: Control the database instance node to sequentially generate and output clone data in the cloning task; wherein, the clone data includes at least tablespace data, data page data that has undergone data changes, and redo log data corresponding to the data page data.

[0030] In this embodiment of the invention, step S30 is used to form a cloned data stream that can be transmitted online, compressed to disk, and reconstructed into recognizable data files on the recovery side. In this embodiment, the cloned data is divided into a tablespace copy stage, a data page copy stage, and a redo log copy stage. The purpose is to separate historical baseline data from changed data within the backup window: the tablespace copy stage provides the historical baseline; the data page copy stage completes the data pages that changed during the tablespace copy; and the redo log copy stage provides physical logs to support the database instance in performing its own recovery replay after startup, thereby achieving consistent recovery. This staged output design allows backup request object nodes to switch write file types according to stages while maintaining data correspondence, reducing the difficulty of reconstruction on the recovery side.

[0031] In an alternative implementation, step S30 includes the following sub-steps: S301: Read the metadata dictionary data of the database system to obtain the tablespace file list; during the tablespace copy phase, traverse the tablespace file list, read the tablespace data sequentially and output it to form the first stage clone data; S302: during the data page copy phase, determine the data page sequence number list of data changes based on the data change records generated during the tablespace copy phase, and read the corresponding data page data and output it to form the second stage clone data; S303: during the redo log copy phase, read the archived redo log data and output it to form the third stage clone data.

[0032] Furthermore, regarding the data page copying phase, data page archiving is initiated before the tablespace copying phase begins to record the sequence numbers of data pages whose data changes occur during the tablespace copying phase. Upon entering the data page copying phase, the data page archiving record is read to form the data page sequence number list, and the data page sequence number list is output all at once. After outputting the data page sequence number list, the index positions corresponding to each data page sequence number in the tablespace file are read sequentially, and the data page data at the index positions is read from the tablespace file and output, until the output of the data page data corresponding to the data page sequence number list is completed.

[0033] Specifically, step S301 obtains the tablespace file list through the metadata dictionary, enabling the tablespace copy to cover the range of historical data that needs to be backed up in the database instance. The tablespace data serves as the base file input for subsequent recovery, and its output order can be executed according to the tablespace file list order, or scheduled according to tablespace size or IO load; no particular order is limited.

[0034] Specifically, step S302 is used to obtain the set of data pages that have changed within the tablespace copy window. Since the database is still running during online backup, some pages read during tablespace copy may be updated during the copy process. Relying solely on tablespace copy will result in gaps in the baseline obtained on the recovery side. Therefore, this step forms a list of data page sequence numbers through data change records and reads the corresponding data page data output according to the sequence number, so that the backup side can obtain the content for page-level repair, thereby replacing the data page at the corresponding position in the tablespace file with the data page data during recovery.

[0035] Specifically, step S303 is used to output redo log data corresponding to the data page changes. The redo log data can be used for log replay and consistency correction after the database instance is started on the target node; it can also serve as supplementary evidence for backup window boundaries to verify the completeness of data page change capture. The redo log can be an archived log or an equivalent persistent log file, and the log organization and storage method are not limited.

[0036] In another optional implementation, step S30 supports parallel cloning: the database instance node starts multiple clone threads to read data from different tablespaces or data pages associated with different tablespaces in parallel and outputs the data. The backup request object node can maintain a one-to-one data receiving relationship with the clone threads to improve the overall backup throughput. The degree of parallelism can be determined by system parameters or user request configuration, without limiting the number of threads or scheduling strategy. Specifically, during the tablespace copying phase, multiple clone threads are started to read data from different tablespaces in parallel and output the data; during the backup request object node, multiple compression threads corresponding to the number of clone threads are started, so that each compression thread receives the data output by the corresponding clone thread and performs compressed writing; during the data page copying phase, multiple clone threads are started to read data from data pages associated with different tablespaces that have undergone data changes in parallel and output the data, while maintaining the corresponding compression thread to perform compressed writing on the output data.

[0037] Step S40: Receive the cloned data, and perform compression processing on the tablespace data, the data page data, and the redo log data respectively based on the compression algorithm configuration parameters, and write the compressed data into the compressed backup data storage directory to form compressed backup data.

[0038] Specifically, step S40 is used to transform the cloned data stream generated online into a persistent, copyable, and decompressible compressed backup data set. In this embodiment, tablespace data, data page data, and redo log data are compressed and written to different types of files respectively. The purpose is to allow for separate processing on the recovery side according to file type: tablespace files are used directly as baseline files to generate database-recognizable files; data page files are used for replacement and repair according to page sequence numbers; and redo log files are used for playback by the database's own mechanism after startup. By compressing and writing data separately, the boundaries between different data types can be maintained during the compression stage, making the processing path on the recovery side clear and less prone to confusion.

[0039] In an alternative implementation, step S40 includes the following sub-steps: S401: In the tablespace copy phase, a tablespace compressed file associated with the tablespace name is created for each tablespace file, and the received tablespace data is compressed and written to the tablespace compressed file. Subsequent received data for the same tablespace is appended to the compressed file. S402: In the data page copy phase, a data page file associated with the tablespace name is created for each tablespace, and the received data page data is encapsulated according to a preset data page file organization format and compressed before being written to the data page file. Subsequent received data for the same tablespace is appended to the compressed file. S403: In the redo log copy phase, the received redo log data is compressed and written to the redo log compressed file.

[0040] Specifically, step S401 is used to establish the association between the tablespace compressed file and the tablespace name, enabling the recovery side to rebuild the corresponding tablespace file according to the tablespace name. The append-only write method allows the tablespace compressed file to continuously receive a continuous stream of data from the cloning task, avoiding the management overhead caused by frequently creating small files.

[0041] Specifically, step S402 is used to establish the association between the data page compressed file, tablespace name, and page sequence number index. Since the data page data is a data patch that has changed during the tablespace copying phase, the recovery side must be able to locate and write the data page content to the corresponding index position in the tablespace file based on the data page sequence number. Therefore, this embodiment introduces a data page file organization format to store the correspondence between page sequence numbers and page content in the data page file, thereby supporting the subsequent recovery algorithm to read and replace the content sequentially. This organization format is not limited to a single implementation form, as long as it can express the page sequence number-page content mapping and support segmented reading.

[0042] Specifically, step S403 is used to generate a compressed redo log file. After being decompressed on the target node, the compressed redo log file can be placed in a log path or archive path that the database instance can recognize, so that it can be read and replayed by the database's own recovery mechanism after the instance starts. This log file may contain all redo logs within the backup window, or it may contain log segments related to the data page capture window. The specific scope can be configured according to the system policy.

[0043] In a preferred embodiment, refer to Figure 2 This document provides an example of a data page file organization format. The data page file organization format includes data blocks and data pages, with the first data page of each data block being an index page. Specifically, in one example implementation, the organization format includes: A data page consists of one or more data blocks; a data block consists of several data pages; the first data page in a data block stores the data block's metadata information; the first field of the data block metadata page is an identifier field, used to identify whether the current data block is the last data block in the data page file; the remaining fields of the data block metadata page are data page sequence numbers. If the data page sequence number is the maximum value, then the field is invalid and no longer associated with a specific data page; except for the first data page, the remaining data pages of a data block are data page data that the database system can recognize.

[0044] Based on this, the data is encapsulated and compressed according to a preset data page file organization format and written to the data page file. Subsequent received data in the same tablespace is appended to the data page file. Specifically, this includes: dividing a data page file into one or more data blocks, each data block containing several data pages, with the first data page being an index page; writing an identifier field in the first field of the index page to identify whether the current data block is the last data block of the data page file; sequentially writing data page sequence numbers in the remaining fields of the index page, and storing the data page content corresponding to the data page sequence number recorded in the second field of the index page in the second data page of the data block, thereby establishing a correspondence between data page sequence numbers and data page content; creating a new data block after the data block is full and appending it to the previous data block, and updating the identifier field when it is determined to be the last data block.

[0045] In practical applications, compressed backup first receives a list of all modified page numbers of a specific tablespace file, and then... Figure 2The data page file organization format shown calculates the maximum number of data pages that can be stored in a single data block. If the number of remaining page sequence numbers is greater than the maximum number of data pages that can be stored in the data block, it is written to the data block metadata page. The identifier field indicates that the data block is not the last data block, and the remaining fields are written to the data page sequence numbers in the list. If the number of remaining page sequence numbers is not greater than the maximum number of data pages that can be stored in the data block, it is written to the data block metadata page. The identifier field indicates that the data block is the last data block, and the remaining fields are written to the data page sequence numbers in the list. If the number of remaining page sequence numbers is less than the maximum number of data pages that can be stored in the data block, after writing the last data page sequence number, a maximum value is written to the fields of the next metadata page. After the metadata page is written, it is compressed, and the sequence number value already written to the metadata page is removed from the page sequence number list. The compressed backup continues to receive data page data, compresses the data pages after receiving them, and writes them to the data page file. After the compressed backup task reads the status change notification sent by the database instance node cloning task, it switches the task status and waits for the redo log copy data.

[0046] Specifically, the identifier field is used to provide the data block end boundary for the recovery algorithm, enabling the recovery side to determine when to end the parsing of the current data block and when to end the parsing of the entire data page file when reading the data page file; and by providing the page sequence index, the recovery side can write the data page content to the correct position in the tablespace file; and ensure that the data page content itself still maintains the database page format, so that the tablespace file after replacement writing has the recognition of the database instance.

[0047] It should be noted that the above organizational format is only an example for illustrative purposes. The implementation of this invention does not limit the number of fields in the index page, the order of field arrangement, or the method of taking the maximum value, as long as it can realize the corresponding mapping between page number and page content and support the location replacement on the recovery side.

[0048] Step S50: Copy the compressed backup data to the target node to be restored, and perform decompression processing on the compressed backup data on the target node using a compression algorithm consistent with the compression algorithm configuration parameters to obtain decompressed backup data.

[0049] Specifically, step S50 is used to transfer the set of compressed files generated on the backup side to the recovery side and restore it into a set of decompressed files that can be used for file recovery processing. Since the compression algorithm configuration parameters are used for compression writing in step S40, reusing the same set of parameters during decompression on the target node can avoid problems such as decompression failure, data corruption, or file length mismatch caused by inconsistent algorithms or parameters. The copying can be achieved through network transmission, offline media, or shared storage; the specific transmission method is not limited.

[0050] In an optional implementation, step S50 includes the following sub-steps: S501: Copy the compressed backup data to the target node to be restored, and provide the compression algorithm configuration parameters as decompression parameters to the data decompression tool; S502: Control the data decompression tool to traverse the compressed backup data directory to obtain all compressed files, and decompress each compressed file to generate the corresponding decompressed file; S503: Delete the compressed file after each compressed file is decompressed, until the entire compressed backup data directory is decompressed.

[0051] It should be noted that this decompression and deletion strategy can be used to release the temporary space of the target node, but whether or not to delete it does not constitute a limitation on the implementation of this invention.

[0052] Step S60: Perform file recovery processing on the decompressed backup data to restore the decompressed backup data to a file format recognizable by the database instance, and start the database instance based on the data file after the file recovery processing is completed to complete the recovery.

[0053] Specifically, step S60 organizes the decompressed tablespace files, data page files, and redo log files into a set of bootable database instance files. The tablespace files provide a historical baseline; the data page files provide patches for changed data pages within the backup window; and the redo log files are used for replay by the database's own recovery mechanism after instance startup, thereby achieving eventual consistency. The core of this step is: first, based on the data page sequence number, the data page content is replaced and written to the corresponding index position in the tablespace file, making the tablespace file recognizable by the database; then, the database instance is started, and the database's own mechanism reads the redo log to complete the recovery.

[0054] In an alternative implementation, step S60 includes the following sub-steps: S601: Traverse the decompressed backup data directory to obtain a list of tablespace files, and check for the existence of a corresponding data page file for each tablespace file; if no corresponding data page file exists, skip the data page replacement process for that tablespace file. S602: If a corresponding data page file exists, read the data page file by data block, and read the identifier field and data page sequence number from the index page of each data block. S603: Obtain the tablespace file index position based on the data page sequence number, read the data page content corresponding to the data page sequence number, and replace the data page data at the index position with the data page content. S604: Determine whether the current data block has been completely read and whether it is the last data block. If not, continue reading the next data page content; if the current data block has been completely read and is not the last data block, continue reading the next data block; if the last data block has been reached, exit data page file reading. S605: After completing the data recovery of the current data page file, delete the data page file. S606: Start the database instance based on the replaced tablespace file, and use the database's own recovery mechanism to read the redo log to complete data recovery.

[0055] Specifically, step S601 establishes the matching relationship between tablespace files and data page files at the start of recovery. Since data page files are patch files generated at the tablespace granularity, if a tablespace has not undergone any changes that need to be captured during the backup window, a corresponding data page file may not exist. In this case, directly using the tablespace file as recovery input can reduce unnecessary processing and improve recovery efficiency. It should be noted that the absence of a corresponding data page file does not mean that the tablespace has not changed; it only means that patching via data page patches is unnecessary. Eventual consistency can still be guaranteed by subsequent redo log replay.

[0056] Specifically, step S602 is used to read the data page file by data block and parse the index page. The index page contains an identifier field and multiple data page sequence number fields, which serve to provide a location index for the data page content: the recovery side can determine which page sequence numbers are contained in the current data block by reading the index page, and thus write the subsequent data page content to the correct location in the tablespace file according to the sequence number. This mechanism allows the recovery side to complete the location without having to parse the page header information in the data page content, reducing the parsing complexity. The reading by data block can also divide the data page file into multiple processing units of controllable size, facilitating sequential reading and anomaly recovery.

[0057] Specifically, step S603 is used to perform page-level replacement writing. In this sub-step, the recovery side calculates or finds the index position in the tablespace file based on the data page sequence number, and overwrites the data page content corresponding to that page sequence number into that position, thereby repairing the pages in the tablespace file that have changed during the backup window. This replacement writing process can directly restore the page consistency of the tablespace file at the data level, making the tablespace file closer to a correct state recognizable by the database after the replacement is completed.

[0058] Specifically, step S604 is used to control the traversal logic and determine the termination boundary using the identifier field. Since each data block contains a limited number of page sequence number fields, when the index page field is full, a new data block needs to be created to continue recording. Therefore, the recovery side needs to determine whether the current data block has been processed and whether it is the last data block.

[0059] In one example implementation, when certain page number fields in the index page reach their maximum value, it indicates that the field is invalid and no longer associated with a specific data page, thus prematurely ending the processing of valid pages for the current data block. When the identifier field indicates that the current data block is the last data block and all valid pages have been processed, the entire data page file reading and recovery process can be terminated. Through this mechanism, the recovery algorithm can complete the boundary identification of the data page file without relying on an external metadata table.

[0060] Specifically, step S605 releases the temporary patch file after page replacement is complete, reducing disk usage; whether to delete it can be configured according to the operation and maintenance policy and is not limited to mandatory deletion. Step S606 starts the database instance and triggers the database's own recovery mechanism to read the redo log and complete the replay. The reason why redo log replay is still needed here is that during online backup, there are factors such as backup window boundaries and concurrent transactions, and simple page replacement may not be able to cover all consistency conditions; reading the redo log through the database's own mechanism can complete transaction consistency and structural consistency correction, thereby completing the final recovery.

[0061] Therefore, this invention proposes a database cloning, compression, backup, and recovery method. By cloning database instance data within the database system and compressing the cloned data in real time, it effectively improves the flexibility of data backup, reduces the complexity of data backup deployment, and utilizes the database's own cloning capabilities to minimize the impact on database operations during backup. Furthermore, and more importantly, it effectively reduces the disk space requirements for database backup. By decompressing the database and utilizing the database itself for database file recovery, the complexity of backup data recovery is effectively reduced.

[0062] Reference Figure 3 , Figure 3 This is a schematic diagram of the database cloning, compression, backup, and recovery system according to an embodiment of the present invention.

[0063] like Figure 3 As shown, the database cloning, compression, backup, and recovery system proposed in this embodiment of the invention includes: Parsing module 10 is used to obtain the user's clone compression backup request statement, and parse it to obtain the backup mode, database instance node connection information, compressed backup data storage directory and compression algorithm configuration parameters; The sending module 20 is used to send a compressed clone request instruction to the database instance node based on the backup mode, so as to trigger the database instance node to start the cloning task during normal database operation. Output module 30 is used to control the database instance node to sequentially generate and output clone data in the cloning task; wherein, the clone data includes at least tablespace data, data page data that has undergone data changes, and redo log data corresponding to the data page data; Compression module 40 is used to receive the cloned data, and perform compression processing on the tablespace data, the data page data and the redo log data respectively based on the compression algorithm configuration parameters, and write the compressed data into the compressed backup data storage directory to form compressed backup data; The decompression module 50 is used to copy the compressed backup data to the target node to be restored, and to perform decompression processing on the compressed backup data on the target node using a compression algorithm consistent with the compression algorithm configuration parameters to obtain decompressed backup data. The recovery module 60 is used to perform file recovery processing on the decompressed backup data to restore the decompressed backup data to a file format recognizable by the database instance, and to start the database instance based on the data file after the file recovery processing is completed to complete the recovery.

[0064] Other embodiments or specific implementations of the database cloning, compression, backup, and recovery system of the present invention can be found in the above-described method embodiments, and will not be repeated here.

[0065] It is understood that in the description of this specification, references to terms such as "one embodiment," "another embodiment," "other embodiments," or "first embodiment to Nth embodiment," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0066] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0067] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A database cloning, compression, backup, and recovery method, characterized in that, Includes the following steps: Obtain the user's clone compression backup request statement, and parse it to obtain the backup mode, database instance node connection information, compressed backup data storage directory, and compression algorithm configuration parameters; Based on the backup mode, a compressed clone request command is sent to the database instance node to trigger the database instance node to start the cloning task during normal database operation. The database instance nodes are controlled to sequentially generate and output clone data in the cloning task; wherein, the clone data includes at least tablespace data, data page data that has undergone data changes, and redo log data corresponding to the data page data; The cloned data is received, and the tablespace data, data page data, and redo log data are compressed based on the compression algorithm configuration parameters. The compressed data is then written to the compressed backup data storage directory to form compressed backup data. The compressed backup data is copied to the target node to be restored, and the compressed backup data is decompressed on the target node using a compression algorithm that is consistent with the compression algorithm configuration parameters to obtain decompressed backup data. Perform file recovery processing on the decompressed backup data to restore the decompressed backup data to a file format recognizable by the database instance, and start the database instance based on the data file after the file recovery processing is completed to complete the recovery.

2. The database cloning, compression, backup, and recovery method as described in claim 1, characterized in that, Based on the backup mode, a compressed clone request command is sent to the database instance node to trigger the database instance node to start the cloning task during normal database operation, specifically including: Based on the backup mode obtained by parsing the user's clone compression backup request statement, determine whether the backup mode is a local clone compression mode or a remote clone compression mode. If the local clone compression mode is determined, the backup request object node and the database instance node are determined to be the same node, and the pre-set compression variables and compression algorithm variables are read to generate the compression algorithm configuration parameters; When the remote cloning compression mode is determined, the database instance node connection information is extracted from the cloning compression backup request statement and a network communication connection is created. The pre-set compression variables and compression algorithm variables are read to generate the compression algorithm configuration parameters. Based on the backup mode and the compression algorithm configuration parameters, a compressed cloning request instruction that can be recognized by the database instance node is constructed.

3. The database cloning, compression, backup, and recovery method as described in claim 1, characterized in that, Controlling the database instance nodes to sequentially generate and output clone data in the cloning task specifically includes: Read the database system's metadata dictionary data to obtain a list of tablespace files; During the tablespace copying phase, the tablespace file list is traversed to read the tablespace data sequentially and output it to form the first phase of cloned data; During the data page copying phase, a list of data page numbers that have undergone data changes is determined based on the data change records generated during the tablespace copying phase. The corresponding data page data is then read and output to form the second-stage cloned data. During the redo log copying phase, the archived redo log data is read and output to form the third-stage cloned data.

4. The database cloning, compression, backup, and recovery method as described in claim 3, characterized in that, During the data page copying phase, a list of data page numbers that have undergone data changes is determined based on the data change records generated during the tablespace copying phase. The corresponding data page data is then read and output. Specifically, this includes: Before the tablespace copy phase begins, start the data page archiving record to record the sequence number of the data page that has undergone data changes during the tablespace copy phase; When entering the data page copying stage, the data page archive records are read to form the data page sequence number list, and the data page sequence number list is output all at once; After outputting the data page number list, the index positions of each data page number in the data page number list in the tablespace file are read sequentially, and the data page data at the index positions is read from the tablespace file and output, until the data page data corresponding to the data page number list is output.

5. The database cloning, compression, backup, and recovery method as described in claim 1, characterized in that, The process involves receiving the cloned data and performing compression processing on the tablespace data, data page data, and redo log data based on the compression algorithm configuration parameters. The compressed data is then written to the compressed backup data storage directory to form compressed backup data. Specifically, this includes: During the tablespace copying phase, a tablespace compressed file associated with the tablespace name is created for each tablespace file, and the received tablespace data is compressed and written to the tablespace compressed file. Subsequent received data for the same tablespace is appended. During the data page copying phase, a data page file associated with the tablespace name is created for each tablespace. The received data page data is encapsulated and compressed according to the preset data page file organization format and written to the data page file. Subsequent received data for the same tablespace is appended to the data page file. During the redo log copying phase, the received redo log data is compressed and written to the redo log compressed file.

6. The database cloning, compression, backup, and recovery method as described in claim 5, characterized in that, The data page file organization format includes: data blocks and data pages, with the first data page of each data block being an index page; after being encapsulated according to a preset data page file organization format, it is compressed and written into the data page file, specifically including: A data page file is divided into one or more data blocks, each data block contains several data pages, and the first data page is the index page; Write an identifier field into the first field of the index page to identify whether the current data block is the last data block of the data page file; The remaining fields of the index page are written with the data page number value in sequence, and the data page content corresponding to the data page number recorded in the second field of the index page is stored in the second data page of the data block, thereby establishing the correspondence between the data page number and the data page content one by one. After a data block is full, a new data block is created and appended to the previous data block, and the identifier field is updated when it is determined to be the last data block.

7. The database cloning, compression, backup, and recovery method as described in claim 1, characterized in that, The method further includes: During the tablespace copying phase, multiple clone threads are started to read data output from different tablespaces in parallel. On the backup request object node, start multiple compression threads corresponding to the number of clone threads, so that each compression thread receives the data output by the corresponding clone thread and performs compression writing. During the data page copying phase, multiple clone threads are started to read and output the data of the data pages associated with different tablespaces that have undergone data changes in parallel, while the corresponding compression thread performs compression writing on the output data.

8. The database cloning, compression, backup, and recovery method as described in claim 1, characterized in that, The compressed backup data is copied to the target node to be restored, and then decompressed on the target node using a compression algorithm consistent with the compression algorithm configuration parameters to obtain decompressed backup data. Specifically, this includes: The compressed backup data is copied to the target node to be restored, and the compression algorithm configuration parameters are provided as decompression parameters to the data decompression tool. The data decompression tool is controlled to traverse the compressed backup data directory to obtain all compressed files, and each compressed file is decompressed to generate the corresponding decompressed file; Delete each compressed file after it has been decompressed, until the entire compressed backup data directory has been decompressed.

9. The database cloning, compression, backup, and recovery method as described in claim 1, characterized in that, Perform file recovery processing on the decompressed backup data to restore it to a file format recognizable by the database instance, and start the database instance based on the data file after file recovery processing to complete the recovery. Specifically, this includes: Read the data page file corresponding to the tablespace, and read the index page of each data block to obtain the identification field and data page number; Based on the data page sequence number, obtain the index position in the corresponding tablespace file, and read the data page content corresponding to the data page sequence number in the data page file; The data page content is used to replace the data page data at the index position in the tablespace file to complete the recovery of the corresponding data page, and this process is repeated until the last data block has been processed is determined according to the identifier field. After replacing all the data pages corresponding to the data page file, delete the data page file. The database instance is started based on the tablespace file after the data pages have been replaced, and the data recovery is completed using the database's own recovery mechanism.

10. A database cloning, compression, backup, and recovery system, characterized in that, include: The parsing module is used to obtain the user's clone compressed backup request statement and parse it to obtain the backup mode, database instance node connection information, compressed backup data storage directory, and compression algorithm configuration parameters. The sending module is used to send a compressed clone request instruction to the database instance node based on the backup mode, so as to trigger the database instance node to start the cloning task during normal database operation; The output module is used to control the database instance nodes to sequentially generate and output clone data in the cloning task; wherein, the clone data includes at least tablespace data, data page data that has undergone data changes, and redo log data corresponding to the data page data; The compression module is used to receive the cloned data, and perform compression processing on the tablespace data, the data page data and the redo log data respectively based on the compression algorithm configuration parameters, and write the compressed data into the compressed backup data storage directory to form compressed backup data; The decompression module is used to copy the compressed backup data to the target node to be restored, and to perform decompression processing on the compressed backup data on the target node using a compression algorithm consistent with the compression algorithm configuration parameters to obtain decompressed backup data. The recovery module is used to perform file recovery processing on the decompressed backup data to restore the decompressed backup data to a file format that the database instance can recognize, and to start the database instance based on the data file after the file recovery processing is completed to complete the recovery.

Images