A cross-region disaster recovery backup method based on snapshot data incremental synchronization

Abstract

The application relates to the technical field of data disaster recovery backup, in particular to a cross-Region disaster recovery backup method based on snapshot data incremental synchronization, which comprises the following steps: acquiring a cluster snapshot descriptor and a metadata mirror file of a current round; judging whether a cluster snapshot descriptor of a previous round exists in object storage of a target region; if not, uploading data files based on the current cluster snapshot descriptor; if yes, determining a partition-level difference set based on the two cluster snapshot descriptors, performing an incremental synchronization operation according to the partition-level difference set and determining a to-be-deleted list; uploading the metadata mirror file and the cluster snapshot descriptor of the current round and marking the cluster snapshot descriptor as an available state; after confirming that the cluster snapshot descriptor is in the available state, initiating a file deletion request based on the to-be-deleted list to delete the files in the to-be-deleted list. The application can improve the cross-Region disaster recovery backup efficiency and safely clean up invalid redundant data.

Description

Technical Field

[0001] This application relates to the field of data disaster recovery and backup technology, and in particular to a cross-Region disaster recovery and backup method based on incremental synchronization of snapshot data. Background Technology

[0002] Currently, in the storage-compute separation architecture, compute nodes and data storage are decoupled. Data files are usually stored in object storage services, and high availability of data is achieved through the multi-replica mechanism provided by the object storage service provider.

[0003] Existing cross-region disaster recovery backup solutions typically employ periodic full backups, backing up object storage data from the source region to the target region at fixed, long intervals—essentially, periodically backing up all data to another region. However, because full backups involve reading massive amounts of data and transferring it across regions, the backup operation is time-consuming and resource-intensive, limiting the backup frequency and making it difficult to meet low recovery point objective (RPO) requirements. Furthermore, full backup solutions struggle to securely identify and clean up expired backup data, leading to an unlimited growth of backup data.

[0004] The existing technical solutions mentioned above have the following drawbacks: the existing cross-region disaster recovery backup methods have low backup efficiency and cannot safely clean up failed and redundant data, so there is room for improvement. Summary of the Invention

[0005] To improve the efficiency of cross-region disaster recovery backup and safely clean up failed and redundant data, this application provides a cross-region disaster recovery backup method based on incremental synchronization of snapshot data.

[0006] The above-mentioned objective of this application is achieved through the following technical solution:

[0007] A cross-Region disaster recovery backup method based on incremental synchronization of snapshot data, the cross-Region disaster recovery backup method includes:

[0008] Obtain the cluster snapshot descriptor and metadata image file for the current round. The cluster snapshot descriptor includes at least the database identifier, data table identifier, data partition identifier, and corresponding version number.

[0009] Determine if the cluster snapshot descriptor from the previous round exists in the object storage of the target region;

[0010] If it does not exist, then based on all data partition identifiers recorded in the cluster snapshot descriptor of the current round, obtain the data files corresponding to all data partition identifiers, and upload the data files from the local object storage to the object storage of the target region;

[0011] If it exists, then based on the cluster snapshot descriptor of the current round and the cluster snapshot descriptor of the previous round, determine the partition-level difference set, perform incremental synchronization operation according to the partition-level difference set and determine the list to be deleted. The partition-level difference set includes the deleted partition set, the newly added partition set and the version change partition set.

[0012] Upload the metadata image file of the current round and the cluster snapshot descriptor to the object storage of the target region, and mark the cluster snapshot descriptor of the current round as available;

[0013] After confirming that the cluster snapshot descriptor for the current round is in an available state, a file deletion request is initiated to the object storage of the target region based on the list to be deleted, so as to delete the files in the list to be deleted.

[0014] By adopting the above technical solution, the partition-level difference set is determined by comparing the cluster snapshot descriptors of the current round with those of the previous round, and incremental synchronization operations are performed based on this partition-level difference set. This reduces the repeated transmission of full data and improves the efficiency of cross-region disaster recovery backup. At the same time, by performing deletion operations based on the list to be deleted only after confirming the availability of the cluster snapshot descriptor, the cleanup of failed and redundant data is delayed, reducing the accumulation of redundant backup data. Furthermore, by marking the snapshot as available only after data synchronization is completed in the current round, the consistency of backup version switching is improved, and the risk of intermediate snapshots being misused in the recovery process during abnormal backup interruptions is reduced. This improves both backup efficiency and storage utilization while enhancing the security of disaster recovery data.

[0015] In a preferred embodiment, this application can be further configured such that obtaining the cluster snapshot descriptor and metadata image file for the current round specifically includes:

[0016] Send a checkpoint acquisition request to the metadata management node of the cluster to obtain the checkpoint identifier, which is used to lock the metadata state at the current moment;

[0017] Based on the checkpoint identifier, the metadata image file is generated locally, and the metadata image file includes at least the complete metadata information of the cluster;

[0018] Obtain metadata topology information from the metadata image file, and construct the cluster snapshot descriptor for the current round based on the metadata topology information.

[0019] By adopting the above technical solution, a descriptor is constructed by generating a metadata image file based on the checkpoint identifier and extracting topology information. This enables the acquired snapshot descriptor and metadata image file to be based on the metadata view corresponding to the same checkpoint, thereby providing a consistent data benchmark for subsequent difference comparisons and avoiding inconsistencies in backup data due to dynamic changes in metadata.

[0020] In a preferred embodiment, this application can be further configured such that: determining whether a cluster snapshot descriptor from the previous round exists in the object storage of the target region specifically includes:

[0021] Send a query request carrying a preset storage path to the object storage of the target region, and receive the query results returned by the object storage of the target region based on the query request;

[0022] Based on the query results, determine whether historical snapshot files exist in the object storage of the target region;

[0023] If the historical snapshot file exists, the historical snapshot file is downloaded from the object storage of the target region, and the cluster snapshot descriptor of the previous round is obtained by parsing the historical snapshot file.

[0024] By adopting the above technical solution, the current backup task round type can be identified by sending a query request to the target area and determining whether a historical snapshot file exists based on the returned result. If a historical snapshot file exists, the cluster snapshot descriptor of the previous round can be obtained by downloading and parsing the file, thereby providing an accurate old version baseline for subsequent incremental difference comparison.

[0025] In a preferred embodiment, this application can be further configured as follows: obtaining the data files corresponding to all data partition identifiers recorded based on the cluster snapshot descriptor of the current round, specifically including:

[0026] Obtain all data partition identifiers and the current version number corresponding to each data partition identifier;

[0027] Based on all the data partition identifiers and the current version number, a metadata retrieval request is sent to the backend storage node, and the bucket metadata returned by the backend storage node based on the metadata retrieval request is received.

[0028] Parse the bucket metadata to obtain the data file referenced in the bucket metadata.

[0029] By adopting the above technical solution, by requesting bucket metadata from the backend storage node based on the partition identifier and the current version number and parsing out the referenced data files, the data files that need to be backed up in the current round can be extracted during the first round of full backup, thereby providing a data foundation for incremental comparison in subsequent rounds.

[0030] In a preferred embodiment, this application can be further configured such that: determining the partition-level difference set based on the cluster snapshot descriptor of the current round and the cluster snapshot descriptor of the previous round specifically includes:

[0031] Obtain the first set of data partition identifiers from the cluster snapshot descriptor of the previous round, and the second set of data partition identifiers from the cluster snapshot descriptor of the current round;

[0032] Compare the first data partition identifier set with the second data partition identifier set;

[0033] Data partition identifiers that exist in the first data partition identifier set but not in the second data partition identifier set are identified as deleted partition identifiers, and the deleted partition identifiers are added to the deleted partition set.

[0034] Data partition identifiers that exist in the second data partition identifier set but not in the first data partition identifier set are identified as new partition identifiers, and the new partition identifiers are added to the new partition set.

[0035] Data partition identifiers that exist simultaneously in both the first data partition identifier set and the second data partition identifier set are determined as coexisting partition identifiers;

[0036] Obtain the first version number of the coexistence partition identifier in the cluster snapshot descriptor of the previous round, and the second version number in the cluster snapshot descriptor of the current round;

[0037] If the first version number is different from the second version number, the corresponding coexistence partition identifier will be included in the version change partition set.

[0038] By adopting the above technical solution, by comparing the set of partition identifiers in the two rounds of snapshot descriptors and verifying the version number, the addition, deletion and change status of data can be identified at the partition level, thereby narrowing the comparison range for subsequent file-level incremental synchronization and reducing repeated scanning of unchanged partitions.

[0039] In a preferred embodiment, this application can be further configured such that: the incremental synchronization operation based on the partition-level difference set and the determination of the list to be deleted specifically include:

[0040] Based on the deleted partition identifier in the deleted partition set, obtain the data file associated with the deleted partition identifier, and add the data file to the list to be deleted;

[0041] Based on the new partition identifier in the new partition set, obtain the data file corresponding to the new partition identifier, and upload the data file to the object storage of the target area;

[0042] Based on the version change partition identifier in the version change partition set, determine the file-level difference set corresponding to the version change partition identifier;

[0043] Based on the file-level difference set, newly added data files and invalid data files are obtained;

[0044] The newly added data file is uploaded to the object storage of the target area, and the invalid data file is added to the list to be deleted.

[0045] By adopting the above technical solution, corresponding file processing actions are performed for three different types of partitions: deleted, added, and version changed. This can transform partition-level differences into corresponding file upload operations and lists of files to be deleted, thereby supporting incremental data synchronization and subsequent cleanup of invalid files, and improving the correspondence between remote backup data and the current snapshot status.

[0046] In a preferred embodiment, this application can be further configured such that: determining the file-level difference set corresponding to the version change partition identifier based on the version change partition identifier in the version change partition set specifically includes:

[0047] Obtain the old version number of the version change partition identifier in the cluster snapshot descriptor of the previous round, and the new version number in the cluster snapshot descriptor of the current round;

[0048] Obtain the corresponding first bucket metadata set and second bucket metadata set based on the old version number and the new version number, respectively;

[0049] Compare the first bucket metadata set with the second bucket metadata set;

[0050] Data files that exist in the second bucket metadata set but not in the first bucket metadata set are identified as the newly added data files;

[0051] Data files that exist in the first bucket metadata set but not in the second bucket metadata set are identified as invalid data files;

[0052] Based on the newly added data file and the invalid data file, the file-level difference set is generated.

[0053] By adopting the above technical solution, and by comparing the bucket metadata of the version change partition under the old version number and the new version number, it is possible to identify the new and invalid status of files at the file level, thereby extracting incremental data and reducing the amount of cross-regional data transmission.

[0054] In a preferred embodiment, this application can be further configured as follows: uploading the metadata image file of the current round and the cluster snapshot descriptor to the object storage of the target region, and marking the cluster snapshot descriptor of the current round as available, specifically includes:

[0055] Upload the metadata image file of the current round to the preset image storage path of the object storage in the target region, and upload the cluster snapshot descriptor to the preset descriptor storage path of the object storage in the target region;

[0056] Based on the preset descriptor storage path, the latest snapshot pointer file is updated in the object storage of the target region so that the latest snapshot pointer file points to the cluster snapshot descriptor;

[0057] Write a status tag file into the object storage of the target region to mark the cluster snapshot descriptor of the current round as the available state through the status tag file.

[0058] By adopting the above technical solution, by uploading the metadata image file and cluster snapshot descriptor to a preset path, updating the latest snapshot pointer file and writing it to the status tag file, the recovery process can locate the snapshot version that has been marked as available based on the snapshot pointer, and reduce the risk of recovery anomalies caused by the exposure of intermediate state snapshots.

[0059] In a preferred embodiment, this application can be further configured as follows: Initiating a file deletion request to the object storage of the target region based on the list to be deleted, in order to delete the files in the list to be deleted, specifically includes:

[0060] Retrieve all data files to be deleted recorded in the list to be deleted;

[0061] Based on the data files to be deleted, a batch file deletion request is sent to the object storage of the target region;

[0062] Receive the execution result returned by the object storage of the target region in response to the batch file deletion request;

[0063] If the execution result is that the deletion fails, a failure log is recorded locally, and the availability of the cluster snapshot descriptor for the current round is maintained.

[0064] By adopting the above technical solution, and by logging when deletion fails without changing the current snapshot's availability status, the data cleanup operation can be separated from the snapshot availability status control, thereby reducing the impact of deletion anomalies on the use of currently recoverable snapshots.

[0065] In a preferred embodiment, this application can be further configured such that: the cluster snapshot descriptor further includes materialized index information, and the cross-Region disaster recovery backup method further includes:

[0066] Obtain the first materialized index information associated with the data table identifier in the cluster snapshot descriptor of the current round, and the second materialized index information associated with the data table identifier in the cluster snapshot descriptor of the previous round;

[0067] Compare the first materialized index information with the second materialized index information;

[0068] If the first materialized index information is different from the second materialized index information, then it is determined that the data table corresponding to the data table identifier has undergone a table structure change;

[0069] Based on the data table identifier corresponding to the data table whose table structure has changed, obtain the first set of data files associated with the data table identifier in the current round of cluster snapshot descriptor, and the second set of data files associated with the cluster snapshot descriptor in the previous round.

[0070] Compare the first set of data files with the second set of data files;

[0071] Data files that exist in the first data file set but not in the second data file set are identified as new data files and uploaded to the object storage of the target area;

[0072] Data files that exist in the second data file set but not in the first data file set are identified as invalid data files and added to the list to be deleted.

[0073] By adopting the above technical solution and comparing materialized index information, it is possible to identify changes in the structure of the data table and perform difference processing based on the table-level file set when the table structure changes. This improves the integrity and accuracy of data synchronization in this scenario and reduces the risk of missing incremental comparisons due to changes in table structure.

[0074] In summary, this application includes at least one of the following beneficial technical effects:

[0075] 1. By comparing the cluster snapshot descriptors of the current round with those of the previous round, a partition-level difference set is determined, and incremental synchronization operations are performed based on this set. This reduces the duplication of full data transmission and improves the efficiency of cross-Region disaster recovery backup. Simultaneously, by confirming the availability of the cluster snapshot descriptor before performing deletion operations based on the list to be deleted, delayed cleanup of failed and redundant data is achieved, reducing the accumulation of redundant backup data. Furthermore, marking the snapshot as available only after data synchronization is completed in the current round helps improve the consistency of backup version switching and reduces the risk of intermediate snapshots being misused in the recovery process during backup interruptions. This improves both backup efficiency and storage utilization while enhancing the security of disaster recovery data.

[0076] 2. By comparing materialized index information, changes in the data table structure can be identified, and when the table structure changes, difference processing can be performed based on the table-level file set, thereby improving the integrity and accuracy of data synchronization in this scenario and reducing the risk of missing incremental comparisons due to changes in table structure. Attached Figure Description

[0077] Figure 1 This is a flowchart illustrating the implementation of a cross-Region disaster recovery backup method based on incremental synchronization of snapshot data in one embodiment of this application.

[0078] Figure 2 This is another implementation flowchart of a cross-Region disaster recovery backup method based on incremental synchronization of snapshot data in one embodiment of this application. Detailed Implementation

[0079] The following embodiments will help those skilled in the art to further understand the function of this application, but do not limit this application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application. These all fall within the protection scope of this application.

[0080] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0081] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0082] The present application will be further described in detail below with reference to the accompanying drawings.

[0083] In one embodiment, such as Figure 1 As shown, this application discloses a cross-Region disaster recovery backup method based on incremental synchronization of snapshot data, which specifically includes the following steps:

[0084] S10. Obtain the cluster snapshot descriptor and metadata image file for the current round. The cluster snapshot descriptor includes at least the database identifier, table identifier, partition identifier, and corresponding version number.

[0085] In this embodiment, the ClusterSnapshotInfo is a metadata file used to record the logical topology of the cluster at the snapshot time, and the metadata image file is a backup file containing all the metadata information of the cluster. By obtaining these two files, a baseline view at the current time can be provided for subsequent data difference comparison. For example, in the StarRocks storage-compute separation architecture, the ClusterSnapshotInfo records the mapping relationship between the database identifier (DatabaseID), table identifier (Table ID), partition identifier (Partition ID) and the corresponding version number in a tree structure, thereby depicting the distribution status of the data at the current time.

[0086] Specifically, when a cross-Region disaster recovery backup task is triggered, the snapshot descriptor file and the corresponding metadata image file generated at the current moment are read from the local disk or memory. The snapshot descriptor is loaded into memory for subsequent parsing and comparison operations, while the metadata image file is temporarily stored in the local queue to be uploaded, thereby providing the corresponding data source file for the current round of backup.

[0087] S20. Determine whether the target region's object storage contains a cluster snapshot descriptor from the previous round.

[0088] In this embodiment, the object storage of the target region refers to the cloud storage service deployed in a remote data center to store backup data. The cluster snapshot descriptor of the previous round refers to the topology record file generated when the backup task was successfully executed last time. By determining whether it exists, it can be determined whether the current backup task is the first full backup or a subsequent incremental backup. For example, if no historical snapshot descriptor is found in the object storage of the Shanghai Region, it means that a full backup needs to be performed.

[0089] Specifically, a probe request for a specific storage path is sent to the object storage service in the remote data center. The response status code returned by the object storage service is used to confirm whether there is a historical backup record under that path. If it exists, the historical record is read into the local memory as the old version benchmark for subsequent comparison. If it does not exist, the current task is marked as the initial full backup mode, thereby providing a basis for judgment for the subsequent execution of the initial backup process or incremental backup process.

[0090] S30. If it does not exist, then based on all data partition identifiers recorded in the cluster snapshot descriptor of the current round, obtain the data files corresponding to all data partition identifiers, and upload the data files from the local object storage to the object storage of the target area.

[0091] In this embodiment, this step is a full data upload, which is a process of synchronizing all physical data files covered in the current cluster snapshot descriptor to the remote location at once. For example, in the first round of backup, all underlying Parquet or ORC format data files belonging to these partitions in the local object storage need to be copied to the off-site storage to ensure that the disaster recovery site has complete recovery capabilities.

[0092] Specifically, it iterates through each data partition identifier recorded in the cluster snapshot descriptor of the current round, locates the corresponding underlying physical file set in the local object storage based on the partition identifier and version number, calls the data transfer interface to copy these physical files in batches to the object storage of the target area according to the original directory hierarchy, and records the list of transferred files after the upload is completed, thereby completing the full upload of the data files corresponding to the current round.

[0093] S40. If it exists, determine the partition-level difference set based on the cluster snapshot descriptor of the current round and the cluster snapshot descriptor of the previous round, perform incremental synchronization operation according to the partition-level difference set and determine the list to be deleted. The partition-level difference set includes the deleted partition set, the newly added partition set and the version change partition set.

[0094] In this embodiment, the partition-level difference set refers to the summary of partition-level changes obtained by comparing the snapshot descriptors of two rounds, and the list to be deleted refers to a list of redundant files that existed in the old snapshot but are no longer needed in the new snapshot. By calculating the difference set and performing incremental synchronization, only the changed data can be transmitted. At the same time, invalid data is managed through the list to be deleted. For example, if the snapshots at time T1 and time T2 are compared and it is found that partition P1 has been deleted, partition P2 has been added, and the version number of partition P3 has changed, then the upload and cleanup of files are only processed for these changed partitions, thereby greatly reducing cross-region network bandwidth consumption and controlling the expansion of backup data.

[0095] Specifically, the cluster snapshot descriptors of the current round are compared layer by layer with those of the previous round to identify data partitions whose status has changed and classify them into the corresponding difference set. For newly added partitions and partitions with version changes, the latest physical files associated with them are extracted and uploaded. For deleted files and invalid files caused by version changes, the corresponding invalid files in their path information are appended to the list to be deleted. In this way, while synchronizing the changed data, redundant data files that need to be cleaned up later are recorded.

[0096] S50. Upload the metadata image file and cluster snapshot descriptor of the current round to the object storage of the target region, and mark the cluster snapshot descriptor of the current round as available.

[0097] In this embodiment, the available status refers to the state in which the snapshot has successfully completed data synchronization and can be identified and used by the disaster recovery process. By uploading the data first and then submitting the metadata and marking it as available, it helps to improve the consistency of backup version switching. For example, if a failure occurs during the data file upload, the incomplete snapshot will not be used by the recovery process because it is not marked as available, thereby avoiding the damage of the backup from overwriting the original valid data.

[0098] Specifically, the metadata image file and cluster snapshot descriptor temporarily stored in the local upload queue are transmitted to the object storage in the target area via the internal network. After confirming that the file upload is complete, the effective flag of the current snapshot is updated in the object storage, so that the snapshot of this round is visible to the recovery process and is in a recoverable state, thereby ensuring that the backup status is only updated after the data is fully synchronized.

[0099] S60. After confirming that the cluster snapshot descriptor of the current round is in an available state, based on the list to be deleted, initiate a file deletion request to the object storage of the target region to delete the files in the list to be deleted.

[0100] In this embodiment, the strategy of cleaning up old data is to perform this step only after confirming that the new snapshot is fully available. By placing the deletion action after the status flag, it can be ensured that the old snapshot data is not destroyed if any part of the backup process fails. For example, if the metadata upload stage fails, since the deletion stage has not been entered, the previous snapshot and its referenced physical data files are still intact, and the cluster can still be restored based on the old snapshot, thereby reducing the risk of abnormal interruption during the backup process affecting existing recoverable snapshots.

[0101] Specifically, the status flags of the current snapshot are monitored. After reading the available status flags, all invalid physical files recorded in the list to be deleted are extracted, and a batch deletion command is sent to the object storage service in the target area. Redundant data blocks that are no longer referenced by any available snapshots are cleaned up, thereby freeing up remote storage space and ensuring the cleanliness of backup data.

[0102] In one embodiment, step S10, namely obtaining the cluster snapshot descriptor and metadata image file for the current round, specifically includes:

[0103] S11. Send a checkpoint acquisition request to the cluster's metadata management node to obtain the checkpoint identifier, which is used to lock the current metadata state.

[0104] In this embodiment, the metadata management node (Frontend, or FE node for short) is the node responsible for maintaining the global metadata of the cluster. The checkpoint ID is the version number used to represent the metadata consistency snapshot at a specific moment. By obtaining this ID, it can be ensured that the metadata read subsequently is in the same transaction view. For example, in StarRocks, a checkpoint ID is requested from the FE node. This ID corresponds to a stable metadata state without any ongoing transactions, thereby avoiding data inconsistency caused by changes in metadata during the backup process.

[0105] Specifically, a checkpoint retrieval request containing a backup task identifier is submitted to the metadata management node via an internal RPC protocol. Upon receiving the request, the metadata management node suspends the garbage collection operation of metadata and generates a globally unique checkpoint version number to return to the caller, thereby providing a time anchor for the subsequent generation of a consistent image.

[0106] S12. Based on the checkpoint identifier, generate a metadata image file locally. The metadata image file includes at least the complete metadata information of the cluster.

[0107] In this embodiment, the metadata image file refers to a binary or JSON format file obtained by exporting the metadata tree in memory. By generating an image based on checkpoints, the complete logical structure of the cluster at the current moment can be captured. For example, the image file contains all database and table creation statements, partition definitions, and bucket information, thereby providing the most basic metadata support for disaster recovery and reconstruction.

[0108] Specifically, the metadata export tool is invoked to read the full metadata of the corresponding version from the storage engine of the metadata management node based on the obtained checkpoint identifier. The database definition, table structure definition and partition version information are serialized and written to the local disk to generate an image file, thereby completing the local backup of the metadata.

[0109] S13. Obtain metadata topology information from the metadata image file and construct the cluster snapshot descriptor for the current round based on the metadata topology information.

[0110] In this embodiment, metadata topology information refers to the hierarchical relationship between metadata objects at all levels. By extracting topology information to construct descriptors, the direct mapping between metadata and physical files can be decoupled, forming lightweight comparison objects. For example, the ID information of the tree structure of Database→Table→Partition / Version can be parsed from the image file and saved separately as a cluster snapshot descriptor, thereby avoiding loading a huge full metadata image when comparing differences and improving comparison efficiency.

[0111] Specifically, the locally generated metadata image file is parsed, the database and table objects within it are traversed, the identifiers of all data partitions under each table and their currently visible version numbers are extracted, and these hierarchical relationships and version information are reorganized into a preset descriptor data structure and persisted for storage, thereby constructing key credentials for incremental comparison.

[0112] In one embodiment, step S20, namely determining whether a cluster snapshot descriptor from the previous round exists in the object storage of the target region, specifically includes:

[0113] S21. Send a query request carrying a preset storage path to the object storage of the target area, and receive the query results returned by the object storage of the target area based on the query request.

[0114] In this embodiment, the preset storage path refers to a pre-agreed remote directory for storing snapshot descriptor files. By sending a query request, it is possible to detect whether a historical backup exists on the remote end. For example, a query request (HEAD) can be sent to the / backup / snapshot_info / directory of the Shanghai Region object storage. The existence of the file can be determined based on the returned HTTP status code, thereby providing a basis for the selection of the subsequent backup mode.

[0115] Specifically, a query request message containing the remote bucket name and descriptor file path is assembled, and the message is sent to the object storage service in the target region via the HTTPS protocol. The response message containing the file existence status or error code returned by the object storage service is received, thereby completing the detection of the remote historical backup status.

[0116] S22. Based on the query results, determine whether there are historical snapshot files in the object storage of the target area.

[0117] In this embodiment, the historical snapshot file refers to the cluster snapshot descriptor file uploaded when the previous backup round was successful. By parsing the query results, it can be determined whether an incremental comparison is needed. For example, if the query result returns a 404 NotFound status, it is determined that there is no historical snapshot file. If it returns a 200 OK status, it is determined that there is a historical snapshot file, thereby achieving accurate determination of the backup round.

[0118] Specifically, the response message returned by the object storage service client is parsed, and the status code or file metadata information is extracted. If the status code indicates that the file does not exist under the preset path, it is determined that there is no historical snapshot on the remote end. If the status code indicates that the file exists and the size is as expected, it is determined that there is a usable historical snapshot on the remote end, thereby providing decision input for the direction of subsequent logical branches.

[0119] S23. If a historical snapshot file exists, download the historical snapshot file from the object storage of the target region, and parse the cluster snapshot descriptor of the previous round based on the historical snapshot file.

[0120] In this embodiment, parsing the cluster snapshot descriptor of the previous round refers to the process of loading the remote historical backup metadata into memory. By downloading and parsing the file, the data distribution status of the previous round can be obtained. For example, the ClusterSnapshotInfo file of the previous round can be downloaded to the local machine and deserialized to extract the partition identifier and version number set, which can then be used as the old version benchmark for the incremental comparison of the current round.

[0121] Specifically, if the existence of the historical snapshot file is determined, a file download request is sent to the object storage of the target area, the remote descriptor file is streamed to the local temporary directory, the temporary file is read and parsed according to the preset format to extract the database identifier, data table identifier, data partition identifier and version number list contained therein, and reconstructed into the previous round snapshot descriptor object in memory, thereby providing a comparison benchmark for subsequent difference calculations.

[0122] In one embodiment, step S30, which involves obtaining the data files corresponding to all data partition identifiers based on all data partition identifiers recorded in the cluster snapshot descriptor of the current round, specifically includes:

[0123] S31. Obtain all data partition identifiers and the current version number corresponding to each data partition identifier.

[0124] In this embodiment, the partition identifier and the current version number refer to the parameter combination that can uniquely locate the data partition and its latest state. By obtaining this combination, the physical data range that needs to be backed up can be accurately indexed. For example, in StarRocks, the parameter pair with Partition ID 10 and Version 4 can be obtained. This parameter pair directly corresponds to the data snapshot of the partition at the current moment, thereby ensuring the consistency of the data point in time in the full backup.

[0125] Specifically, the tree structure recorded in the cluster snapshot descriptor of the current round is traversed, the data partition identifier and the latest version number field corresponding to the partition on all leaf nodes are extracted, and the extracted parameter pair set is cached in the memory list, thereby providing accurate index conditions for subsequent requests for data from storage nodes.

[0126] S32. Based on all data partition identifiers and the current version number, send a metadata retrieval request to the backend storage node and receive the bucket metadata returned by the backend storage node based on the metadata retrieval request.

[0127] In this embodiment, the backend storage node (BE node) refers to the node responsible for managing the physical storage of data under the storage-compute separation architecture. The bucket metadata (tablet metadata) refers to the information describing the composition of the data files within the bucket. By requesting the metadata of a specific version from the BE node, the list of files that actually need to be backed up can be obtained. For example, by requesting the Tablet metadata of Partition ID 10 under Version 4 from the BE node, the BE node will return a list containing file paths such as file_001.parquet and file_002.parquet, thereby providing a basis for the extraction of physical files.

[0128] Specifically, the partition identifier and version number pair in the memory list are encapsulated into a batch metadata query request. The request is sent to the corresponding backend storage node through an internal communication protocol. The backend storage node searches for the corresponding data file reference list in the local metadata index based on the version information in the request, and returns the found bucket metadata to the caller, thereby obtaining the complete physical file index required for full backup.

[0129] S33. Parse the bucket metadata and obtain the data files referenced in the bucket metadata.

[0130] In this embodiment, the referenced data file refers to the underlying physical file that actually stores user data, recorded in the bucket metadata. By parsing the metadata and extracting these files, the actual data carrier that needs to be uploaded can be obtained. For example, the specific file path and file name on the underlying object storage can be parsed from the Tablet metadata, thereby providing a direct operation object for subsequent data transfer across Regions.

[0131] Specifically, the received bucket metadata is deserialized and parsed, the data file reference field in the metadata is traversed, the complete path identifier and verification information of each physical data file in the local object storage are extracted, and the extracted file path set is deduplicated and used as the data source to be uploaded for the current round of full backup, thereby ensuring the integrity of the backup data.

[0132] In one embodiment, step S40, which involves determining the partition-level difference set based on the cluster snapshot descriptor of the current round and the cluster snapshot descriptor of the previous round, specifically includes:

[0133] S41. Obtain the first data partition identifier set in the cluster snapshot descriptor of the previous round, and the second data partition identifier set in the cluster snapshot descriptor of the current round.

[0134] In this embodiment, the first data partition identifier set and the second data partition identifier set represent the summaries of the identity identifiers of all partitions in the cluster at two different backup times. By extracting these two sets, the complex topology comparison can be transformed into set operations. For example, the partition ID set A in the snapshot at time T1 and the partition ID set B in the snapshot at time T2 can be extracted, thus laying the foundation for subsequent identification of partition-level changes through difference and intersection calculations.

[0135] Specifically, the cluster snapshot descriptors of the previous round and the current round are traversed respectively, and all data partition identifiers are extracted and put into two independent memory collection containers. The two collection containers are deduplicated to construct the first data partition identifier set and the second data partition identifier set for set comparison.

[0136] S42. Compare the first data partition identifier set with the second data partition identifier set.

[0137] In this embodiment, comparing two sets refers to identifying the existence state of elements in different sets through set operations. By comparing, the addition, deletion and coexistence states of partitions can be clearly distinguished. For example, performing intersection and difference operations on set A and set B can classify all partitions into different difference sets according to their state changes.

[0138] Specifically, the first data partition identifier set and the second data partition identifier set are loaded into the memory comparison engine. For each element in the first set, it is checked whether it exists in the second set. At the same time, for each element in the second set, it is checked whether it exists in the first set, thereby identifying the differences and similarities between the two sets.

[0139] S43. Determine the data partition identifiers that exist in the first data partition identifier set but not in the second data partition identifier set as deleted partition identifiers, and add the deleted partition identifiers to the deleted partition set.

[0140] In this embodiment, the deleted partition set refers to a container that records partitions deleted between two backup intervals. By identifying these partitions, it is possible to determine which historical backup data has become invalid. For example, if partition ID 12 exists at time T1 but does not exist at time T2, it will be included in the deleted partition set, thereby providing a basis for subsequent cleanup of redundant backup data.

[0141] Specifically, the difference between the first set of data partition identifiers and the second set of data partition identifiers is calculated. Each data partition identifier in the difference set is traversed, marked as deleted, and appended to the set of deleted partitions, thereby completing the classification of deleted groups.

[0142] S44. Data partition identifiers that exist in the second data partition identifier set but not in the first data partition identifier set are identified as new partition identifiers, and the new partition identifiers are added to the new partition set.

[0143] In this embodiment, the newly added partition set refers to a container that records the newly created partitions between two backup intervals. By identifying these partitions, it can be determined which partitions' data needs to be uploaded in full. For example, if partition ID 15 does not exist at time T1 but exists at time T2, it will be included in the newly added partition set, thereby providing an upload target for subsequent incremental backups.

[0144] Specifically, the difference between the second set of data partition identifiers and the first set of data partition identifiers is calculated. Each data partition identifier in the difference set is traversed, marked as newly added, and appended to the set of newly added partitions, thereby completing the classification of the newly added groups.

[0145] S45. Data partition identifiers that exist simultaneously in the first data partition identifier set and the second data partition identifier set are determined as coexisting partition identifiers.

[0146] In this embodiment, the coexistence partition identifier refers to the partition identity identifier that exists in both backup times. By filtering out coexistence partitions, the comparison range can be further narrowed. Only these partitions are checked to see if their internal versions have changed. For example, if partition ID 10 exists in both set A and set B, it is extracted as the coexistence partition identifier, thereby providing a candidate set for subsequent version-level difference determination.

[0147] Specifically, the intersection of the first set of data partition identifiers and the second set of data partition identifiers is calculated. Each data partition identifier in the intersection is traversed, marked as coexisting, and cached in the coexisting partition list, thereby providing input for subsequent version number comparison.

[0148] S46. Obtain the first version number of the coexistence partition identifier in the cluster snapshot descriptor of the previous round, and the second version number in the cluster snapshot descriptor of the current round.

[0149] In this embodiment, the first version number and the second version number represent the data iteration status of the same partition at different snapshot times. By obtaining these two version numbers, it is possible to determine whether changes such as data import or compression have occurred inside the partition. For example, by obtaining Version 2 of coexisting partition ID 10 at time T1 and Version 4 at time T2, the changes in data inside the partition can be confirmed by the difference in version numbers.

[0150] Specifically, each identifier in the coexistence partition list is traversed, and the version number field corresponding to the identifier is searched in the cluster snapshot descriptor of the previous round and the current round, respectively. The old version number found is recorded as the first version number, and the new version number found is recorded as the second version number, thus forming a version number pair for subsequent comparison.

[0151] S47. If the first version number is different from the second version number, the corresponding coexistence partition identifier will be included in the version change partition set.

[0152] In this embodiment, the version change partition set refers to a coexisting partition container that records the incrementing version number. By determining the different version numbers, partitions whose internal data has changed can be accurately identified. For example, if the version number of partition ID 10 changes from 2 to 4, it will be included in the version change partition set, thereby providing a target range for subsequent file-level incremental comparison.

[0153] Specifically, the extracted version number pairs are compared one by one to determine whether the first version number and the second version number are equal. If they are not equal, it means that the data of the partition has changed between the two snapshots. The status of the coexisting partition is updated to version change and moved into the version change partition set, thereby completing the fine classification of partition-level differences.

[0154] In one embodiment, step S40, which involves performing incremental synchronization based on the partition-level difference set and determining the list to be deleted, specifically includes:

[0155] S48. Based on the deleted partition identifier in the deleted partition set, obtain the data file associated with the deleted partition identifier and add the data file to the list to be deleted.

[0156] In this embodiment, the associated data file refers to the physical file uploaded to the remote end by the deleted partition in the previous backup. By adding these files to the list to be deleted, it can be marked that they are redundant data that can be cleaned up. For example, based on the deleted partition ID 12, all historical backup data blocks of it in the remote object storage can be found, and the corresponding files can be added to the list to be deleted, thereby avoiding invalid data from continuously occupying storage space.

[0157] Specifically, the process iterates through the set of deleted partitions, extracts the physical data file path referenced by each deleted partition from the snapshot descriptor of the previous round, and appends these file paths to the list data structure to be deleted in memory, so that these no longer needed remote files can be cleaned up in a unified manner in subsequent processes.

[0158] S49. Based on the new partition identifier in the new partition set, obtain the data file corresponding to the new partition identifier, and upload the data file to the object storage of the target area.

[0159] In this embodiment, the data file corresponding to the newly added partition refers to the full physical file generated in the local object storage of the newly created partition. By uploading these files, the new data can be synchronized to the remote disaster recovery site. For example, for the newly added partition ID 15, all its underlying data files are extracted from the local storage and transmitted to the target area, thereby ensuring the cross-regional redundancy of the new data.

[0160] Specifically, the newly added partition set is traversed, and the associated physical data files are obtained from the current round snapshot descriptor and backend storage nodes based on the identifier of each newly added partition. The data transfer interface is then called to upload these files in batches to the object storage of the target area, thereby completing the cross-regional synchronization of the newly added partition data.

[0161] S410. Based on the version change partition identifier in the version change partition set, determine the file-level difference set corresponding to the version change partition identifier.

[0162] In this embodiment, the file-level difference set refers to the list of added and invalid physical files in the same partition under different versions. By determining this set, the partition-level version changes can be refined into file-level operation guidelines. For example, for partitions with version changes, by comparing the bucket metadata under the old and new versions, it can be determined which files are added and which files are invalid, thereby realizing incremental data extraction.

[0163] Specifically, for each identifier in the version change partition set, its bucket metadata under the old version and the new version is obtained. By comparing the list of physical files referenced in the metadata of the two versions, the newly added physical files and the physical files that are no longer referenced are identified. These two parts of files are combined into a file-level difference set, thereby providing a basis for subsequent incremental uploads and deletions.

[0164] S411. Based on the file-level difference set, obtain newly added data files and invalid data files.

[0165] In this embodiment, newly added data files refer to physical files introduced in the new version that do not exist in the old version, and invalid data files refer to physical files that exist in the old version but are no longer referenced in the new version. By extracting these two types of files, the specific operation objects of incremental synchronization can be clearly identified. For example, file_008 that needs to be uploaded and file_001 that needs to be deleted can be separated from the file-level difference set, so that different processing logic can be executed respectively.

[0166] Specifically, the file-level difference set is parsed, and the newly added file records contained therein are extracted into the list of newly added data files, and the invalid file records contained therein are extracted into the list of invalid data files, thereby transforming the difference information into directly operable data objects.

[0167] S412. Upload the newly added data file to the object storage of the target area, and add the invalid data file to the list to be deleted.

[0168] In this embodiment, uploading new files and recording invalid files are the core actions of incremental synchronization. By transmitting new data to the remote end and marking invalid data as to be deleted (for example, uploading only file_008 and adding file_001 to the list to be deleted), backup efficiency is greatly improved while ensuring data consistency.

[0169] Specifically, the data transfer component is invoked to read the files in the newly added data file list from the local object storage and write them to the object storage of the target area. At the same time, the file paths in the invalid data file list are appended to the list to be deleted, thereby completing the incremental synchronization and redundancy marking of the partition data of the version change.

[0170] In one embodiment, step S410, namely determining the file-level difference set corresponding to the version change partition identifier based on the version change partition identifier in the version change partition set, specifically includes:

[0171] S4101. Obtain the old version number of the version change partition identifier in the cluster snapshot descriptor of the previous round, and the new version number in the cluster snapshot descriptor of the current round.

[0172] In this embodiment, the old version number and the new version number refer to the version identifiers of the same partition that changed between two snapshots. By obtaining these two version numbers, the specific data snapshot that needs to be compared can be located. For example, the old version number 2 and the new version number 4 of partition ID 10 can be obtained, thereby providing an index for the subsequent extraction of metadata under these two versions.

[0173] Specifically, based on the partition identifier of the version change, the version field corresponding to the partition is queried in the cluster snapshot descriptor of the previous round and the current round, respectively. The historical version found is recorded as the old version number, and the current version found is recorded as the new version number, thereby obtaining the specific value of the version.

[0174] S4102. Obtain the corresponding first bucket metadata set and second bucket metadata set based on the old version number and the new version number, respectively.

[0175] In this embodiment, the first bucket metadata set and the second bucket metadata set represent the physical file reference status of the partition under the old version and the new version, respectively. By obtaining these two sets, the changes between versions can be mapped to the differences in specific file lists. For example, obtaining Tablet metadata set A under Version 2 and Tablet metadata set B under Version 4 can provide a comparison source for file-level difference calculation.

[0176] Specifically, the old version number and partition identifier are combined as query parameters to request historical bucket metadata from the backend storage node, and the new version number and partition identifier are combined as query parameters to request current bucket metadata. The two batches of returned metadata are then assembled into the first bucket metadata set and the second bucket metadata set, respectively, thereby completing the data preparation for file-level comparison.

[0177] S4103. Compare the metadata set of the first bucket with the metadata set of the second bucket.

[0178] In this embodiment, comparing the two bucket metadata sets refers to comparing the differences in the physical file reference lists under the two versions. By comparing, the file-level replacements caused by data import or compression operations can be accurately identified. For example, by comparing the file paths in set A and set B, it can be found out which files are newly added and which files are replaced.

[0179] Specifically, all referenced physical file paths in the first bucket metadata set are extracted to form the old file set, and all referenced physical file paths in the second bucket metadata set are extracted to form the new file set. A comparison operation is performed on the old file set and the new file set to identify the differences between the two sets.

[0180] S4104. Data files that exist in the second bucket metadata set but not in the first bucket metadata set are identified as newly added data files.

[0181] In this embodiment, newly added data files refer to physical files newly generated during the data evolution process. By comparing the sets, the targets that need to be incrementally uploaded can be clearly identified. For example, file_008, which exists in the new file set but not in the old file set, is a newly added data file, thereby providing a precise transmission target for incremental data synchronization.

[0182] Specifically, the difference between the new file set and the old file set is calculated, and each file path in the difference is traversed, marked as newly added, and added to the list of newly added data files, thereby completing the identification of newly added physical files.

[0183] S4105. Data files that exist in the first bucket metadata set but not in the second bucket metadata set are identified as invalid data files.

[0184] In this embodiment, invalid data files refer to old physical files that are no longer referenced by the latest version after data compression or overwriting. By filtering out these files, a basis can be provided for cleaning up backup data. For example, file_001, which exists in the old file set but not in the new file set, is an invalid data file, and can be safely reclaimed in subsequent processes.

[0185] Specifically, the difference between the old file set and the new file set is calculated, and each file path in the difference is traversed, marked as invalid, and added to the list of invalid data files, thereby completing the identification of invalid physical files.

[0186] S4106. Generate a file-level difference set based on newly added data files and invalid data files.

[0187] In this embodiment, the file-level difference set is a summary of newly added and invalid files. By generating this set, version changes can be transformed into a list of directly executable file operations. For example, the newly added file_008 and the invalid file_001 can be combined into a difference record, thereby providing a unified data structure for subsequent incremental uploads and delayed deletions.

[0188] Specifically, the list of newly identified data files and the list of invalid data files are merged into the same data structure, and the data structure is labeled with the corresponding version change partition identifier. This data structure is then output as the file-level difference set of that partition, thereby completing the mapping of version changes to file-level operations.

[0189] In one embodiment, step S50 involves uploading the metadata image file and cluster snapshot descriptor of the current round to the object storage of the target region, and marking the cluster snapshot descriptor of the current round as available. Specifically, this includes:

[0190] S51. Upload the metadata image file of the current round to the preset image storage path of the object storage in the target region, and upload the cluster snapshot descriptor to the preset descriptor storage path of the object storage in the target region.

[0191] In this embodiment, the preset image storage path and the preset descriptor storage path refer to isolated directories in the remote object storage specifically used to store metadata and topology descriptions. By uploading both to the specified paths, it can be ensured that the required files can be retrieved by type during backup and recovery. For example, the metadata image file is uploaded to the / backup / image / directory, and the ClusterSnapshotInfo is uploaded to the / backup / info / directory, thereby achieving standardized management of backup data.

[0192] Specifically, the upload interface of the object storage SDK is called to stream the locally generated metadata image file to the image storage bucket corresponding to the target region. At the same time, the cluster snapshot descriptor file is transferred to the corresponding descriptor storage bucket. After the upload is completed, the ETag is verified to ensure that the file has not been damaged during the transmission, thereby ensuring the reliability of the metadata backup.

[0193] S52. Based on the preset descriptor storage path, update the latest snapshot pointer file in the object storage of the target area so that the latest snapshot pointer file points to the cluster snapshot descriptor.

[0194] In this embodiment, the latest snapshot pointer file refers to an index file that records the location of the latest available snapshot. By updating this pointer, the recovery process can quickly locate the latest backup version. For example, a latest_snapshot pointer file can be maintained in object storage, and its contents can be updated to the path of the current round descriptor. In the event of a disaster, the recovery program only needs to read this pointer to obtain the latest data view.

[0195] Specifically, the complete storage path of the current round cluster snapshot descriptor in the remote object storage is obtained, and this path is written as content to the latest snapshot pointer file, so that the pointer file always points to the snapshot descriptor that was successfully uploaded most recently, thereby achieving a smooth switch of the backup data baseline.

[0196] S53. Write a status tag file to the object storage of the target region to mark the cluster snapshot descriptor of the current round as available through the status tag file.

[0197] In this embodiment, the status flag file refers to a flag file used to identify the integrity of a snapshot transaction. By writing to this file, it can serve as a status identifier that the current round of snapshots has been committed, thereby ensuring the logical integrity of the backup commit process. For example, a SUCCESS flag file is only written after the data file, metadata file, and pointers have all been updated. Once the recovery program detects the flag file, it will recognize that the snapshot is usable, thereby avoiding the use of corrupted backups in intermediate states.

[0198] Specifically, after confirming that the metadata image file, cluster snapshot descriptor, and latest snapshot pointer have all been successfully persisted in the object storage of the target region, a status flag file with a preset name is written to the corresponding snapshot directory on the remote end. The generation of this file indicates that the current round of snapshots has met the conditions for recovery, thereby completing the atomic commit of the backup process.

[0199] In one embodiment, step S60, which involves initiating a file deletion request to the object storage of the target region based on the list of files to be deleted, specifically includes:

[0200] S61. Retrieve all data files to be deleted recorded in the list of files to be deleted.

[0201] In this embodiment, the data files to be deleted refer to redundant physical files that are confirmed to no longer be referenced by any historical version after the latest snapshot is available. By extracting these files, it is possible to prepare the objects for batch cleanup operations. For example, the file paths of all failed partitions and the old file paths generated by version changes can be parsed from the list to be deleted, thereby providing a target list for releasing remote storage space.

[0202] Specifically, the system reads the list of files to be deleted, either in memory or locally, iterates through all records in the list, extracts the complete path identifier of the physical data file corresponding to each record in the remote object storage, and summarizes these path identifiers into a list of files to be deleted, thus providing data input for subsequent batch deletion requests.

[0203] S62. Based on the data files to be deleted, send a batch file deletion request to the object storage of the target area.

[0204] In this embodiment, a batch file deletion request is an instruction submitted to the object storage service to clean up multiple files at once. Sending batch requests can improve the efficiency of cleaning up redundant data. For example, calling the object storage's Batch Delete API can send hundreds of invalid files for deletion at the same time.

[0205] Specifically, the list of files to be deleted is paginated and assembled according to the interface restrictions of the object storage service, generating a batch deletion request message that conforms to the protocol specifications. This message is then sent to the object storage service in the target region via HTTPS, triggering a remote reclamation operation on redundant data blocks, thereby efficiently cleaning up backup data junk.

[0206] S63. Receive the execution results returned by the object storage of the target region for the batch file deletion request.

[0207] In this embodiment, the execution result refers to the object storage service's processing status feedback on the deletion command. By receiving this result, it can be confirmed whether redundant data has been successfully cleaned up. For example, receiving the response message returned by the object storage service, which contains the success or failure status code of each file deletion operation, thus providing a basis for subsequent exception handling.

[0208] Specifically, the system listens for the batch deletion response message returned by the storage service client, parses the message to obtain the status code of each file deletion operation contained therein, and if the status code indicates success, it confirms that the corresponding file has been cleaned up; if the status code indicates failure, it records the path of the corresponding file and the error information, thereby completing the collection of deletion operation results.

[0209] S64. If the execution result is that the deletion fails, then record the failure log locally and maintain the availability of the cluster snapshot descriptor for the current round.

[0210] In this embodiment, recording failure logs and maintaining an available state refers to the fault tolerance strategy adopted when the deletion operation encounters an anomaly. For example, if the deletion of some redundant files fails due to momentary network jitter in the remote object storage, only the log is recorded locally without rolling back the snapshot state. These remaining redundant files can be asynchronously cleaned up through subsequent garbage collection mechanisms, thereby ensuring that the availability of core backup data is not affected by secondary cleanup operations.

[0211] Specifically, if a file record that failed to be deleted is found when parsing the execution result, the file path and corresponding error code are written to the local exception log file. At the same time, the availability status flag of the current round of cluster snapshot descriptor remains unchanged, so that the recovery process can still use the snapshot normally, thereby reducing the impact of file cleanup exceptions on the availability status of the current snapshot.

[0212] In one embodiment, such as Figure 2 As shown, the cluster snapshot descriptor also includes materialized index information, and the cross-Region disaster recovery backup method also includes:

[0213] S71. Obtain the first materialized index information associated with the data table identifier in the cluster snapshot descriptor of the current round, and the second materialized index information associated with the data table identifier in the cluster snapshot descriptor of the previous round.

[0214] In this embodiment, materialized index information refers to metadata used to characterize the physical storage structure and sorting method of a data table. By obtaining materialized index information from two rounds of snapshots, it is possible to detect whether the table structure has changed. For example, in StarRocks, materialized indexes correspond to the data distribution and sorting key definition of a table. Obtaining materialized index information under the same data table identifier from two rounds of snapshots provides a comparison source for identifying changes at the table structure level.

[0215] Specifically, based on the data table identifier, the corresponding materialized index fields are searched in the cluster snapshot descriptors of the current round and the previous round, respectively. The materialized index structure of the current round is extracted as the first materialized index information, and the materialized index structure of the previous round is extracted as the second materialized index information, thereby completing the data preparation for table structure comparison.

[0216] S72. Compare the first materialized index information with the second materialized index information.

[0217] In this embodiment, comparing two materialized index information means checking whether the physical storage structure definition of the table has changed. By comparing, it is possible to find structural changes caused by data definition language (DDL) operations such as modifying the table structure (Alter Table). For example, compare whether the column definitions and sort keys of the same data table in the old and new snapshots are consistent, thereby determining whether special data synchronization processing is required for changes in the table structure.

[0218] Specifically, the column count, column type, and sort key definitions in the first and second materialized index information are compared field by field. If any difference is found in any field, it is determined that the physical storage structure of the data table has changed, thereby identifying the table objects that need to be compared and synchronized based on the table-level file set.

[0219] S73. If the first materialized index information is different from the second materialized index information, then it is determined that the data table corresponding to the data table identifier has undergone a change in table structure.

[0220] In this embodiment, a change in table structure refers to an irreversible reconstruction of the physical definition of the data table. By identifying such a change, a full data backup logic for the table can be triggered. For example, when a new column is added to the data table or the sorting key is modified, the original bucketing and file organization methods are no longer applicable. The file set must be re-extracted based on the new structure to avoid missing data in incremental comparison.

[0221] Specifically, when the first materialized index information is found to be inconsistent with the second materialized index information, the data table is marked as having a structural change status. The regular partition-level incremental comparison of the table is suspended, and instead it is included in the table structure change processing flow, thereby ensuring the accuracy of data synchronization.

[0222] S74. Based on the data table identifier corresponding to the data table whose table structure has changed, obtain the first set of data files associated with the data table identifier in the current round of cluster snapshot descriptor, and the second set of data files associated with the cluster snapshot descriptor in the previous round.

[0223] In this embodiment, the first data file set and the second data file set represent the full physical files associated with the table before and after the table structure change. By obtaining these two sets, the incremental comparison at the partition level can be bypassed, and the difference calculation of the file sets can be performed directly at the table level. For example, the file set A referenced by all partitions of the table under the new structure and the file set B referenced by all partitions of the table under the old structure can be extracted, thereby providing a full-dimensional comparison source for table structure change scenarios.

[0224] Specifically, based on the identifier of the data table whose table structure has changed, all partitions under the table are traversed in the current round of cluster snapshot descriptor to extract all referenced physical file paths to form the first data file set. At the same time, all partitions under the table are traversed in the previous round of cluster snapshot descriptor to extract all referenced physical file paths to form the second data file set, thereby completing the construction of the table-level file view.

[0225] S75. Compare the first data file set with the second data file set.

[0226] In this embodiment, comparing two file sets refers to calculating the differences in physical files under the old and new table structures. By comparing them, new files and obsolete files generated due to table structure reorganization can be identified. For example, the difference and intersection of set A and set B can be calculated to separate the newly added data that needs to be synchronized and the historical redundancy that needs to be cleaned up.

[0227] Specifically, the first set of data files and the second set of data files are loaded into the memory comparison engine. For each file path in the first set, it is checked whether it exists in the second set. At the same time, for each file path in the second set, it is checked whether it exists in the first set, thereby identifying the new and invalid parts between the two sets of files.

[0228] S76. Data files that exist in the first data file set but not in the second data file set are identified as new data files and uploaded to the object storage of the target area.

[0229] In this embodiment, the newly added data file refers to a physical file introduced under the new table structure that does not exist in the old table structure. By uploading it, it can be ensured that the new data after the table structure change is synchronized to the remote disaster recovery site. For example, the data file generated by the addition of a column is uploaded to the off-site object storage, thereby ensuring that the latest table structure data can be obtained during disaster recovery backup.

[0230] Specifically, the difference between the first set of data files and the second set of data files is calculated. Each file path in the difference is traversed and marked as a new data file. The data transfer interface is called to read these files from the local object storage and write them to the object storage of the target area, thereby completing the incremental data synchronization in the scenario of table structure change.

[0231] S77. Data files that exist in the second data file set but not in the first data file set are identified as invalid data files and added to the list to be deleted.

[0232] In this embodiment, invalid data files refer to physical files that exist under the old table structure but are no longer referenced under the new table structure. By adding them to the list to be deleted, these data blocks that have become obsolete due to changes in the table structure can be marked for subsequent cleanup. For example, old format files that are no longer referenced due to changes in the sort key can be added to the cleanup queue, thereby preventing backup data from expanding indefinitely due to changes in the table structure.

[0233] Specifically, the difference between the second set of data files and the first set of data files is calculated. Each file path in the difference is traversed, its status is marked as invalid data file, and these file paths are appended to the list data structure to be deleted in memory. This allows for the unified cleanup of redundant data generated by table structure reorganization in the subsequent delayed deletion process.

[0234] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0235] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A cross-Region disaster recovery backup method based on incremental synchronization of snapshot data, characterized in that, The cross-Region disaster recovery backup method includes: Obtain the cluster snapshot descriptor and metadata image file for the current round. The cluster snapshot descriptor includes at least the database identifier, data table identifier, data partition identifier, and corresponding version number. Determine if the cluster snapshot descriptor from the previous round exists in the object storage of the target region; If it does not exist, then based on all data partition identifiers recorded in the cluster snapshot descriptor of the current round, obtain the data files corresponding to all data partition identifiers, and upload the data files from the local object storage to the object storage of the target region; If it exists, then based on the cluster snapshot descriptor of the current round and the cluster snapshot descriptor of the previous round, determine the partition-level difference set, perform incremental synchronization operation according to the partition-level difference set and determine the list to be deleted. The partition-level difference set includes the deleted partition set, the newly added partition set and the version change partition set. Upload the metadata image file of the current round and the cluster snapshot descriptor to the object storage of the target region, and mark the cluster snapshot descriptor of the current round as available; After confirming that the cluster snapshot descriptor for the current round is in an available state, a file deletion request is initiated to the object storage of the target region based on the list to be deleted, so as to delete the files in the list to be deleted.

2. The cross-Region disaster recovery backup method according to claim 1, characterized in that, The process of obtaining the cluster snapshot descriptor and metadata image file for the current round specifically includes: Send a checkpoint acquisition request to the metadata management node of the cluster to obtain the checkpoint identifier, which is used to lock the metadata state at the current moment; Based on the checkpoint identifier, the metadata image file is generated locally, and the metadata image file includes at least the complete metadata information of the cluster; Obtain metadata topology information from the metadata image file, and construct the cluster snapshot descriptor for the current round based on the metadata topology information.

3. The cross-Region disaster recovery backup method according to claim 1, characterized in that, The determination of whether the target region's object storage contains a cluster snapshot descriptor from the previous round specifically includes: Send a query request carrying a preset storage path to the object storage of the target region, and receive the query results returned by the object storage of the target region based on the query request; Based on the query results, determine whether historical snapshot files exist in the object storage of the target region; If the historical snapshot file exists, the historical snapshot file is downloaded from the object storage of the target region, and the cluster snapshot descriptor of the previous round is obtained by parsing the historical snapshot file.

4. The cross-Region disaster recovery backup method according to claim 1, characterized in that, The process of obtaining the data files corresponding to all data partition identifiers based on the cluster snapshot descriptor record of the current round specifically includes: Obtain all data partition identifiers and the current version number corresponding to each data partition identifier; Based on all the data partition identifiers and the current version number, a metadata retrieval request is sent to the backend storage node, and the bucket metadata returned by the backend storage node based on the metadata retrieval request is received. Parse the bucket metadata to obtain the data file referenced in the bucket metadata.

5. The cross-Region disaster recovery backup method according to claim 1, characterized in that, The determination of the partition-level difference set based on the cluster snapshot descriptor of the current round and the cluster snapshot descriptor of the previous round specifically includes: Obtain the first set of data partition identifiers from the cluster snapshot descriptor of the previous round, and the second set of data partition identifiers from the cluster snapshot descriptor of the current round; Compare the first data partition identifier set with the second data partition identifier set; Data partition identifiers that exist in the first data partition identifier set but not in the second data partition identifier set are identified as deleted partition identifiers, and the deleted partition identifiers are added to the deleted partition set. Data partition identifiers that exist in the second data partition identifier set but not in the first data partition identifier set are identified as new partition identifiers, and the new partition identifiers are added to the new partition set. Data partition identifiers that exist simultaneously in both the first data partition identifier set and the second data partition identifier set are determined as coexisting partition identifiers; Obtain the first version number of the coexistence partition identifier in the cluster snapshot descriptor of the previous round, and the second version number in the cluster snapshot descriptor of the current round; If the first version number is different from the second version number, the corresponding coexistence partition identifier will be included in the version change partition set.

6. The cross-Region disaster recovery backup method according to claim 5, characterized in that, The step of performing incremental synchronization operations based on the partition-level difference set and determining the list to be deleted specifically includes: Based on the deleted partition identifier in the deleted partition set, obtain the data file associated with the deleted partition identifier, and add the data file to the list to be deleted; Based on the new partition identifier in the new partition set, obtain the data file corresponding to the new partition identifier, and upload the data file to the object storage of the target area; Based on the version change partition identifier in the version change partition set, determine the file-level difference set corresponding to the version change partition identifier; Based on the file-level difference set, newly added data files and invalid data files are obtained; The newly added data file is uploaded to the object storage of the target area, and the invalid data file is added to the list to be deleted.

7. The cross-Region disaster recovery backup method according to claim 6, characterized in that, The step of determining the file-level difference set corresponding to the version change partition identifier based on the version change partition identifier in the version change partition set specifically includes: Obtain the old version number of the version change partition identifier in the cluster snapshot descriptor of the previous round, and the new version number in the cluster snapshot descriptor of the current round; Obtain the corresponding first bucket metadata set and second bucket metadata set based on the old version number and the new version number, respectively; Compare the first bucket metadata set with the second bucket metadata set; Data files that exist in the second bucket metadata set but not in the first bucket metadata set are identified as the newly added data files; Data files that exist in the first bucket metadata set but not in the second bucket metadata set are identified as invalid data files; Based on the newly added data file and the invalid data file, the file-level difference set is generated.

8. The cross-Region disaster recovery backup method according to claim 1, characterized in that, The step of uploading the metadata image file of the current round and the cluster snapshot descriptor to the object storage of the target region, and marking the cluster snapshot descriptor of the current round as available, specifically includes: Upload the metadata image file of the current round to the preset image storage path of the object storage in the target region, and upload the cluster snapshot descriptor to the preset descriptor storage path of the object storage in the target region; Based on the preset descriptor storage path, the latest snapshot pointer file is updated in the object storage of the target region so that the latest snapshot pointer file points to the cluster snapshot descriptor; Write a status tag file into the object storage of the target region to mark the cluster snapshot descriptor of the current round as the available state through the status tag file.

9. The cross-Region disaster recovery backup method according to claim 1, characterized in that, The step of initiating a file deletion request to the object storage of the target region based on the list of files to be deleted specifically includes: Retrieve all data files to be deleted recorded in the list to be deleted; Based on the data files to be deleted, a batch file deletion request is sent to the object storage of the target region; Receive the execution result returned by the object storage of the target region in response to the batch file deletion request; If the execution result is that the deletion fails, a failure log is recorded locally, and the availability of the cluster snapshot descriptor for the current round is maintained.

10. The cross-Region disaster recovery backup method according to claim 7, characterized in that, The cluster snapshot descriptor also includes materialized index information, and the cross-Region disaster recovery backup method further includes: Obtain the first materialized index information associated with the data table identifier in the cluster snapshot descriptor of the current round, and the second materialized index information associated with the data table identifier in the cluster snapshot descriptor of the previous round; Compare the first materialized index information with the second materialized index information; If the first materialized index information is different from the second materialized index information, then it is determined that the data table corresponding to the data table identifier has undergone a table structure change; Based on the data table identifier corresponding to the data table whose table structure has changed, obtain the first set of data files associated with the data table identifier in the current round of cluster snapshot descriptor, and the second set of data files associated with the cluster snapshot descriptor in the previous round. Compare the first set of data files with the second set of data files; Data files that exist in the first data file set but not in the second data file set are identified as new data files and uploaded to the object storage of the target area; Data files that exist in the second data file set but not in the first data file set are identified as invalid data files and added to the list to be deleted.

Images