Data lake table tape backup method and device, computer equipment and storage medium

Abstract

The invention discloses a data lake table tape backup method and device, computer equipment and a storage medium. The method comprises the following steps: scanning a table list to be backed up to judge whether a table needing to be backed up exists or not; when the table needing to be backed up exists, the table to be backed up is determined; performing backup according to the type of the table to be backed up; and if the table cannot be obtained, writing all records from the last snapshot backup into a new Iceberg table for all tables involved in the backup, copying related data to a specified tape directory, and inserting the corresponding records into a metadata table. By implementing the method disclosed by the invention, economical and reliable long-term data backup can be realized, and the method is particularly suitable for application scenes with strict requirements on data persistence and integrity so as to solve the problems of efficiency and management of backup of a big data system.

Description

Technical Field

[0001] This invention relates to data backup methods, and more specifically to data lake table tape backup methods, apparatus, computer equipment, and storage media. Background Technology

[0002] With the widespread adoption of Hadoop, enterprises have accumulated massive amounts of relational data in their HDFS data lakes, leading to a growing demand for efficient and reliable data lake table backup solutions. This demand is particularly pronounced for critical data requiring long-term storage of over ten years and the ability to be restored to snapshots of any date.

[0003] Currently, the mainstream data lake table backup strategy involves building a dedicated backup cluster and using file copying tools such as distcp to copy newly added directories and files to this cluster. However, this method has significant drawbacks. First, it has high maintenance costs, including hardware purchases, software licensing, hiring professional personnel, and daily operating expenses. Second, there are issues with metadata and data consistency. Table data not only includes the actual data files but also involves metadata stored in databases such as MySQL. Because this metadata is stored differently from the data files, they usually require different backup strategies, increasing management complexity and the risk of inconsistency. Third, historical snapshot recovery is difficult. Existing methods struggle to achieve data recovery at any point in time, which is a major shortcoming for backing up critical data.

[0004] Therefore, it is necessary to design a new method to achieve both economical and reliable long-term data backup, which is particularly suitable for application scenarios with strict requirements for data durability and integrity, in order to solve the efficiency and management problems of big data system backup. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a data lake table tape backup method, apparatus, computer equipment and storage medium.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a data lake table tape backup method, comprising:

[0007] Scan the list of tables to be backed up to determine if the tables can be retrieved;

[0008] If the table can be obtained, determine whether there is a table to be backed up in the table;

[0009] If there are tables to be backed up, back them up according to the type of the tables to be backed up;

[0010] If the tables cannot be obtained, for all tables involved in this backup, all records since the last snapshot backup will be written into a new Iceberg table, the relevant data will be copied to the specified tape directory, and the corresponding records will be inserted into the metadata table.

[0011] The further technical solution is as follows: determining whether there is a table to be backed up in the table includes:

[0012] For table queries, the metadata table is used to obtain the last backup time. If the current backup is the first one, the last backup time is set to zero.

[0013] Get the most recent modification time of the table;

[0014] Determine whether the most recent modification time is not greater than the last backup time;

[0015] If the most recent modification time is not greater than the last backup time, then the scan of the list of tables to be backed up is performed to determine whether there are any tables that need to be backed up.

[0016] If the most recent modification time is greater than the last backup time, then the table is determined to be a table to be backed up.

[0017] The further technical solution is as follows: the backup based on the type of the table to be backed up includes:

[0018] When the type is Base, copy the entire snapshot's data and metadata file, and record it to the metadata table;

[0019] When the type is Daily backup, only the data files newly added and the updated metadata compared to the last snapshot are backed up and recorded in the metadata table;

[0020] When the type is Merge backup, multiple consecutive Daily backups are merged into one Merge backup, and the metadata is updated.

[0021] The further technical solution is as follows: when the type is Base, the entire snapshot's data and metadata file are copied and recorded in the metadata table, including:

[0022] When the type is Base, all data files, manifest files, manifest list files, and metadata files in the entire snapshot are copied to the specified tape directory, and relevant information is recorded in the metadata table.

[0023] The further technical solution is as follows: When the type is Daily backup, only the data files newly added and the updated metadata compared to the last snapshot are backed up, and recorded in the metadata table, including:

[0024] When the type is Daily backup, compare the differences between the current snapshot and the last backup snapshot, back up only the newly added data files and their related manifest, manifest list and metadata files, and add new records to the metadata table.

[0025] The further technical solution is as follows: after backing up the table according to its type, the process includes:

[0026] When restoring the table, look up the tape information where the backup for the target date was located in the metadata table;

[0027] Mount the specified tape according to the tape information, and copy the metadata of the target date to the specified location;

[0028] Metadata records are sorted in reverse order of backup date, and data files are copied from the corresponding tapes to the destination in sequence.

[0029] The further technical solution is as follows: the metadata records are arranged in reverse order of backup date, and the data files are copied sequentially from the corresponding tape to the destination, including:

[0030] Read relevant records about metadata from the specified location and arrange them in reverse order to obtain sorted metadata records;

[0031] Iterate through the sorted metadata records and copy the data files corresponding to the dates from the corresponding tapes to the destination.

[0032] The present invention also provides a data lake table tape backup device, comprising:

[0033] The scanning unit is used to scan the list of tables to be backed up and determine whether the tables can be obtained.

[0034] The judgment unit is used to determine whether a table to be backed up exists in the table if the table can be obtained.

[0035] A backup unit is used to back up the table according to its type.

[0036] The update unit is used to write all records since the last snapshot backup into a new Iceberg table for all tables involved in this backup if the table cannot be obtained, copy the relevant data to the specified tape directory, and insert the corresponding records into the metadata table.

[0037] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described method.

[0038] The present invention also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0039] The advantages of this invention compared to existing technologies are as follows: By intelligently scanning the list of tables to be backed up and performing the appropriate level of backup operation only when a table is detected, this invention effectively improves backup efficiency and management convenience. This method not only avoids unnecessary resource waste but also ensures that the latest state of data is properly preserved. For cases where no tables are to be backed up, all change records since the last snapshot are automatically summarized, integrated into a new Iceberg table, and this critical data is securely copied to a designated tape storage directory. Simultaneously, the metadata table is updated to reflect the latest backup status. This method greatly optimizes the backup strategy of big data systems, improves the efficiency of data processing and recovery, and provides a solid guarantee for data-intensive applications.

[0040] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating the data lake surface tape backup method provided in an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of a sub-process of the data lake surface tape backup method provided in an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of the backup principle provided in an embodiment of the present invention;

[0045] Figure 4 A flowchart illustrating a data lake surface tape backup method provided in another embodiment of the present invention;

[0046] Figure 5 A schematic block diagram of a data lake surface tape backup device provided in an embodiment of the present invention;

[0047] Figure 6 A schematic block diagram of a data lake surface tape backup device provided in another embodiment of the present invention;

[0048] Figure 7A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0051] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0052] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0053] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating the data lake table tape backup method provided in this embodiment of the invention. This data lake table tape backup method is applied to a server. It intelligently scans and determines whether tables need backup, executes corresponding backup strategies based on table type (Base, Daily, or Merge), and records detailed metadata information in a metadata table, thereby achieving efficient and differentiated long-term backup management of tables in the data lake. This method not only ensures data persistence and integrity but also optimizes storage usage by merging consecutive Daily backups into Merge backups, reducing the storage cost of redundant data. During the recovery process, based on the detailed backup information provided by the metadata table, data files are accurately restored in reverse chronological order according to the backup date, greatly improving the efficiency and reliability of big data system backup and recovery. It is particularly suitable for application scenarios with strict requirements for data security and availability.

[0054] Figure 1 This is a flowchart illustrating the data lake surface tape backup method provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110 to S140.

[0055] S110. Scan the list of tables to be backed up and determine if the tables can be obtained.

[0056] The main purpose of this step is to identify all data tables in the system that may require backup. This is done by scanning a list of tables in the database or metadata store to determine which tables are likely to contain newly added or modified data and therefore require backup.

[0057] First, the system will start a scanning process, which will access the database or metadata management system (such as Iceberg table format) to read all table information that may have changed.

[0058] For each scanned table, the system will record its basic information, including but not limited to key identifiers such as the table name and the name of the database it belongs to. This information is used in subsequent steps to determine whether the table needs to be backed up.

[0059] Specifically, scan the list of tables that need to be backed up. If a next table exists, let's say the table name is t and the database name is d. If no next table exists, proceed to S140; otherwise, execute step S120.

[0060] S120. If the table can be obtained, determine whether there is a table to be backed up in the table.

[0061] In this embodiment, the table to be backed up refers to the table that needs to be backed up.

[0062] Identify which tables among all the scanned tables have changed since the last backup and truly require a backup. This step aims to improve backup efficiency and avoid unnecessary resource consumption.

[0063] In one embodiment, please refer to Figure 2 The above-mentioned step S120 may include steps S121 to S124.

[0064] S121. For the table query metadata table, obtain the last backup time. If the current backup is the first one, set the last backup time to zero.

[0065] For each table that needs to be backed up, the system queries the metadata table to obtain the timestamp of its last backup. If this is the first time the table is being backed up, the default last backup time is set to zero.

[0066] S122. Obtain the most recent modification time of the table.

[0067] In this embodiment, the most recent modification time of the table is obtained. This information can usually be obtained directly from the database management system, and it reflects the specific point in time when the table was last modified.

[0068] S123. Determine whether the most recent modification time is not greater than the last backup time;

[0069] If the most recent modification time is not greater than the last backup time, then proceed to step S110;

[0070] Compare the two timestamps (last modified time and last backup time). If the last modified time is not later than the last backup time (i.e., last modified time ≤ last backup time), it means that the table has not changed since the last backup and does not need to be backed up again.

[0071] If the result indicates that no backup is needed, proceed to step S110 to check the next table.

[0072] S124. If the most recent modification time is greater than the last backup time, then the table is determined to be a table to be backed up.

[0073] When a table is found to have been last modified later than the last backup time, it means that the table has new or updated data and should be included in the current backup plan. At this time, the system marks the table as "pending backup" and prepares to perform the corresponding backup operation according to the specific backup strategy (such as Daily, Merge, or Base).

[0074] Through the above steps, the method of this embodiment provides an efficient way to filter out the data tables that truly need to be backed up. This not only improves the overall efficiency of the backup process but also ensures the consistency and integrity of the data, making it particularly suitable for large-scale data management and long-term backup needs.

[0075] S130. When there is a table to be backed up, back it up according to the type of the table to be backed up.

[0076] In this embodiment, when the type is Base, the data and metadata files of the entire snapshot are copied and recorded in the metadata table; specifically, when the type is Base, all data files, manifest files, manifestlist files and metadata files of the entire snapshot are copied to the specified tape directory and the relevant information is recorded in the metadata table.

[0077] When the type is Daily backup, only the newly added data files and updated metadata compared to the last snapshot are backed up and recorded in the metadata table; specifically, when the type is Daily backup, the differences between the current snapshot and the last backup snapshot are compared, and only the newly added data files and their related manifest, manifestlist and metadata files are backed up, and new records are added to the metadata table.

[0078] When the type is Merge backup, multiple consecutive Daily backups are merged into one Merge backup, and the metadata is updated.

[0079] Please see Figure 3 The method in this embodiment combines three different backup strategies: Daily, Merge, and Base. These backup methods aim to reduce costs, simplify metadata management, and ensure data and metadata consistency.

[0080] Daily backup refers to incremental backup of newly added data every day. The i-th day is denoted as Di.

[0081] content:

[0082] New data files: usually stored in Parquet format, containing new table data added on the same day.

[0083] Full manifest file: This is a manifest file that records the location and information of all related data files.

[0084] The manifestlist file consists of multiple manifest files, corresponding to a snapshot of the Iceberg table.

[0085] Metadata file: Records all information about Iceberg table snapshots, including all historical snapshots.

[0086] Location: These files are all located in the Iceberg table directory and were backed up to tape according to the native Iceberg directory structure.

[0087] Merge backups are created by combining multiple consecutive Daily backups. For example, if we merge five consecutive Daily backups into a single Merge backup, denoted as Mi+5, it will contain all the data files from Di+2 to Di+5, as well as the metadata file for Di+5.

[0088] Contents: Data Files: Contains all data files from the merged Daily.

[0089] The most recent Daily's metadata file: that is, the latest manifest file, manifestlist file, and metadata file.

[0090] This backup method reduces the number of Daily backups that need to be processed separately, while retaining the latest metadata information, making recovery operations easier.

[0091] The base backup represents a complete snapshot, containing all the data files and their corresponding metadata files for the Iceberg table at a specific point in time.

[0092] Contents: Data file: Includes the full data file of the Iceberg table.

[0093] Manifest files, manifestlist files, and metadata files: These files provide detailed metadata information, supporting quick and accurate recovery of the entire table's state.

[0094] Function: Base backup provides a solid foundation for long-term data protection, ensuring effective data recovery even over long periods of time.

[0095] By combining Base, Merge, and Daily backup methods, this invention can efficiently restore table snapshot data from any date. The backup frequency is controlled by setting parameters k and r.

[0096] After every k Daily backups are generated, a Merge backup is performed.

[0097] After every r merge backups are generated, a base backup is performed. In this way, in the worst case, data recovery can be completed by accessing only k+r-1 tapes (including 1 base backup, r-1 merge backups, and k-1 daily backups), thus ensuring high data recovery efficiency and good fault tolerance.

[0098] To manage and track all backup activities, an Iceberg table called `meta` was introduced to record information related to all historical backups. The specific columns of this table are as follows:

[0099] db: Indicates the name of the database to which the backup table belongs.

[0100] tab: Identifies the table name for the backup table.

[0101] type: Indicates the backup type, with values ​​of 'D' (Daily), 'M' (Merge), and 'B' (Base).

[0102] dt: Records the data time corresponding to the backup.

[0103] snapid: Identifies the snapshot ID corresponding to the backup.

[0104] dev: Marks the tape device identifier used for backup.

[0105] In this way, the system can effectively manage and retrieve backup information for any time period, ensuring data security and recoverability.

[0106] S140. If the tables cannot be obtained, for all tables involved in this backup, write all records since the last snapshot backup into a new Iceberg table, copy the relevant data to the specified tape directory, and insert the corresponding records into the metadata table.

[0107] In this embodiment, step S140 relates to the operation process when no table to be backed up exists. This step primarily addresses how, after backing up all tables to be processed, all records from the last snapshot (Base) backup of all tables involved in this backup process are written into a new Iceberg table, the relevant data is copied to a specified tape directory, and the corresponding records are inserted into the metadata table. The specific operations are as follows:

[0108] First, for all tables involved in this backup, the metadata table needs to be queried to determine all backup records for these tables since the last Base backup. This step is achieved by executing a specific SQL query that filters out records that meet certain criteria.

[0109] WITH R1 AS: This section is used to filter out all relevant records from all tables involved in this backup.

[0110] WITH R2 AS: Next, find the time point of the last Base backup, that is, the time of the last Base backup of each table before the date of this backup.

[0111] The final SELECT statement combines the results from the two temporary tables above to select all backup records from the last Base backup to the current backup date.

[0112] Based on the obtained data, create a new Iceberg table named meta.current, and write the filtered records into this new table. This step includes not only copying the data files, but also processing the related manifest files, manifest list files, and metadata files.

[0113] Copy data files: Copy all data files within the selected time period to the / backup_dt / meta / current / data subdirectory under the tape directory.

[0114] Copy metadata files: Similarly, copy the manifest file, manifest list file, and metadata file to the / backup_dt / meta / current / metadata subdirectory.

[0115] The final step is to update the metadata table by inserting a new record to reflect the backup operation on the meta.current table. This record contains the following information:

[0116] Database name (db);

[0117] The table name (tab) is, in this case, meta.current;

[0118] Backup type (type), here should be 'B' to indicate Base backup;

[0119] The backup data time (dt) is the specific date of this backup.

[0120] Snapshot ID (snapid) represents the snapshot identifier corresponding to this backup;

[0121] Tape device identifier (dev);

[0122] By performing the steps described above, S140 ensures that all historical backup information can be effectively managed and tracked even when no new tables are available for backup, further enhancing the overall system's data recovery capabilities and management efficiency. This method is particularly suitable for scenarios requiring long-term storage of large amounts of data, such as data storage needs in the financial industry, providing a cost-effective and easy-to-manage data backup solution.

[0123] In this embodiment, the entire method is as follows: First, the system scans the list of tables that need to be backed up. If a next table to be processed is found, let's assume the table name is 't' and its database name is 'd'. If no more tables need to be backed up, the process proceeds to the step of updating the metadata table.

[0124] For each table t, the system queries the metadata table to obtain the last backup time (last_dt). If no record related to table t is found in the metadata table, it means this is the first backup of the table, and in this case, last_dt = 0. Next, the system checks the last modification time of table t (last_modify). If last_modify is less than or equal to last_dt, it indicates that the table has not undergone any recent changes, so the current table is skipped and the process continues scanning the list of tables that need to be backed up.

[0125] Determine the backup type based on the following conditions:

[0126] If this is the first time backing up table t, set the backup type to 'B' (Base) and jump to the Base backup step.

[0127] If the current backup uses the same tape device as the last one, and the device still has enough free space, then the backup type is set to 'D' (Daily), and the process jumps to the Daily backup step.

[0128] Check if the number of Daily backups since the last Merge backup is less than k-1. If so, set the backup type to 'D' (Daily) and jump to the Daily backup step.

[0129] Check if the number of Merge backups since the last Base backup is less than r-1. If so, set the backup type to 'M' (Merge) and jump to the Merge backup step.

[0130] In all other cases, the backup type is set to 'B' (Base), and the process jumps to the Base backup step.

[0131] For each Base backup, copy all data files, manifest files, manifest list files, and metadata files from the most recent snapshot S to the corresponding subdirectories under the tape directory / backup_dt / db / t / (data files are placed in the data subdirectory, and metadata files are placed in the metadata subdirectory). Then, insert a new record into the metadata table containing information for (db, backup_dt, 'B', backup_dt, S, dev).

[0132] In Daily Backup, the snapshot S' corresponding to the last backup is first located in the metadata table. By comparing the manifest and manifest list files of S and S', the newly added data file is identified and copied to the tape directory / backup_dt / db / t / data. Simultaneously, the manifest, manifest list, and metadata files of S are copied to / backup_dt / db / t / metadata. Finally, a new record (db, t, 'D', backup_dt, S, dev) is inserted into the metadata table.

[0133] During the Merge backup process, first locate the snapshot S' corresponding to the most recent Merge or Base backup, and the set of all snapshots {S1, ..., Sn} after S'. For any two consecutive snapshots Si and Sj, identify the newly added data files by comparing their manifest files and manifest list files, and copy them to the tape directory / backup_dt / db / t / data. Subsequently, copy the manifest file, manifest list file, and metadata file of S to / backup_dt / db / t / metadata. Insert a new record (db, t, 'M', backup_dt, S, dev) into the metadata table.

[0134] The process involves updating the metadata table. After backing up all tables, for all tables involved in this backup, the metadata table is queried to retrieve all backup records since the last Base backup, and these records are written to a new Iceberg table, `meta.current`. This process involves a complex SQL query to filter out records that meet certain criteria and write these records to the new Iceberg table. Finally, all data files, manifest files, manifest list files, and metadata files from the latest snapshot of `meta.current` are copied to the corresponding subdirectories under the tape directory ` / backup_dt / meta / current / `, and the corresponding records (meta, current, 'B', backup_dt, S, dev) are inserted into the metadata table. This ensures that all historical backup information is accurately recorded, facilitating subsequent data recovery and management.

[0135] The aforementioned data lake table tape backup method effectively improves backup efficiency and management convenience by intelligently scanning the list of tables to be backed up and performing the appropriate level of backup operation only when a table is detected, based on its type. This method not only avoids unnecessary resource waste but also ensures that the latest state of the data is properly preserved. For cases where no tables need to be backed up, all change records since the last snapshot are automatically summarized, integrated into a new Iceberg table, and this critical data is securely copied to the designated tape storage directory. Simultaneously, the metadata table is updated to reflect the latest backup status. This method greatly optimizes the backup strategy of big data systems, improves the efficiency of data processing and recovery, and provides a solid guarantee for data-intensive applications.

[0136] Figure 4 This is a flowchart illustrating a data lake surface tape backup method according to another embodiment of the present invention. Figure 4 As shown, the data lake table tape backup method of this embodiment includes steps S210-S270. Steps S210-S240 are similar to steps S110-S140 in the above embodiment and will not be described again here. The following details the additional steps S250-S270 in this embodiment.

[0137] S250. When restoring the table, look up the tape information where the backup for the target date is located in the metadata table.

[0138] In this embodiment, tape information refers to the device identifier and storage path of the tape containing the data snapshot associated with a specific backup date, which is recorded in the metadata table.

[0139] Specifically, when a user wants to restore table dt to a specific target date target_dt and specifies the file system directory as / dest, the first step is to query the metadata table for backup information related to that target date. This step involves locating the tape device identifier (e.g., / dev) that contains the data snapshot for the target date. The metadata table, as a core component, records details of each backup operation, including but not limited to key information such as the backup date (dt) and the tape device used (dev). Therefore, by querying the metadata table, the tape containing the data backup corresponding to the target date target_dt can be precisely located.

[0140] S260. Mount the specified tape according to the tape information and copy the metadata of the target date to the specified location.

[0141] In this embodiment, once the tape device identifier for storing backup data on the target date is determined, the next step is to mount the corresponding tape device based on this information. After successful mounting, the target date-specific metadata file (e.g., / target_dt / meta / current) on the tape needs to be copied to the specified destination directory (e.g., / dest / meta / current). This step ensures that subsequent steps can be performed based on the latest metadata and also verifies the accuracy of the selected tape and backup date.

[0142] S270. Arrange metadata records in reverse order of backup date, and copy data files from the corresponding tapes to the destination in sequence.

[0143] In this embodiment, relevant records about metadata are read from the specified location and arranged in reverse order to obtain sorted metadata records;

[0144] Iterate through the sorted metadata records and copy the data files corresponding to the dates from the corresponding tapes to the destination.

[0145] After completing the above steps, the next step is to read the metadata records for the table db.t from the destination directory and sort these records in reverse order by the backup date (dt) field. The reason for using this reverse order strategy is to prioritize restoring the most recent data version to ensure data integrity and consistency. Subsequently, the sorted metadata records are traversed. For each record, based on its backup date (backup_dt) and tape device identifier (backup_dev), the corresponding data file for that date is copied from the corresponding tape to the final destination (e.g., / dest / db / t). This sequential processing method not only improves the efficiency of data recovery but also ensures that even in the event of partial data loss, as much information as possible can be recovered from previous backups.

[0146] In summary, this embodiment provides an economical and reliable long-term data backup and recovery solution, particularly suitable for applications requiring high data durability and integrity. By storing data and metadata in a standardized data lake format, combining the low-cost advantages of tape storage media, and incorporating incremental backup technology to reduce redundancy, this method significantly improves the backup efficiency and data recovery capabilities of big data systems.

[0147] The method in this embodiment aims to solve the problems of format incompatibility, software unavailability, and high cost in traditional backup solutions, while improving data recovery efficiency and capabilities. The main features of this method are as follows:

[0148] Long-term backup reliability: By using standard open-source data lake formats for data and metadata backup, data accessibility is ensured for at least ten years, avoiding data inaccessibility issues caused by outdated formats or software.

[0149] Cost-effective: Data backup solutions based on tape storage not only reduce the need for dedicated hardware and maintenance personnel, but also lower energy costs, providing a more economical backup option.

[0150] Optimized data redundancy and recovery efficiency: An incremental backup strategy is adopted to reduce data redundancy and improve backup performance. This method requires only k+r-1 tapes to complete data recovery, ensuring high efficiency and fault tolerance in the data recovery process.

[0151] Robust data recovery capabilities: Supports data snapshot recovery from any date, making it particularly suitable for industries that need to preserve large amounts of historical data for extended periods, such as the financial sector. Magnetic tape, as a low-cost, long-life, and easy-to-maintain storage medium, performs exceptionally well in these scenarios.

[0152] Combining modern data lake technologies: As modern data lake table formats such as Iceberg gradually become standards, their table definition evolution and snapshot capabilities bring new opportunities for data backup. Our proposed solution, based on tape storage and the Iceberg table format, enables incremental backup of data and metadata, effectively reducing management complexity and ensuring consistency.

[0153] In summary, this method integrates Base, Merge, and Daily backup methods to provide an economical and efficient long-term data backup solution, addressing the main challenges of existing solutions and improving the overall level of data management and protection.

[0154] Figure 5 This is a schematic block diagram of a data lake table tape backup device 300 provided in an embodiment of the present invention. Figure 5 As shown, corresponding to the above data lake table tape backup method, the present invention also provides a data lake table tape backup apparatus 300. This data lake table tape backup apparatus 300 includes a unit for performing the above-described data lake table tape backup method, and the apparatus can be configured in a server. Specifically, please refer to... Figure 5 The data lake surface tape backup device 300 includes a scanning unit 301, a judgment unit 302, a backup unit 303, and an update unit 304.

[0155] The scanning unit 301 is used to scan the list of tables to be backed up and determine whether the tables can be obtained.

[0156] The judgment unit 302 is used to determine whether there is a table to be backed up in the table if the table can be obtained.

[0157] Backup unit 303 is used to back up the table according to its type;

[0158] Update unit 304 is used to write all records since the last snapshot backup into a new Iceberg table for all tables involved in this backup if the table cannot be obtained, copy the relevant data to the specified tape directory, and insert the corresponding records into the metadata table.

[0159] In one embodiment, the determination unit 302 includes:

[0160] The first acquisition subunit is used to query the metadata table of the table to obtain the last backup time, wherein if the current backup is the first one, the last backup time is set to zero; the second acquisition subunit is used to obtain the most recent modification time of the table; the time comparison subunit is used to determine whether the most recent modification time is not greater than the last backup time; if the most recent modification time is not greater than the last backup time, the scanning of the list of tables to be backed up is performed to determine whether there are any tables that need to be backed up; the determination subunit is used to determine that the table is a table to be backed up if the most recent modification time is greater than the last backup time.

[0161] In one embodiment, the backup unit 303 is used to copy the entire snapshot's data and metadata files and record them to the metadata table when the type is Base.

[0162] When the type is Daily backup, only the data files newly added and the updated metadata compared to the last snapshot are backed up and recorded in the metadata table;

[0163] When the type is Merge backup, multiple consecutive Daily backups are merged into one Merge backup, and the metadata is updated.

[0164] In one embodiment, the backup unit 303 is used to copy all data files, manifest files, manifest list files and metadata files of the entire snapshot to a specified tape directory when the type is Base, and record relevant information in the metadata table.

[0165] In one embodiment, the backup unit 303 is used to compare the differences between the current snapshot and the last backup snapshot when the type is Daily backup, back up only the newly added data files and their related manifest, manifest list and metadata files, and add new records to the metadata table.

[0166] Figure 6 This is a schematic block diagram of a data lake surface tape backup device 300 provided in another embodiment of the present invention. Figure 6 As shown, the data lake table tape backup device 300 in this embodiment is based on the above embodiment with the addition of an information search unit 305, a copy unit 306, and a sorting unit 307.

[0167] The information lookup unit 305 is used to look up the tape information of the backup of the target date in the metadata table when restoring the table; the copy unit 306 is used to mount the specified tape according to the tape information and copy the metadata of the target date to the specified location; the sorting unit 307 is used to sort the metadata records in reverse order of the backup date and copy the data files from the corresponding tapes to the destination in sequence.

[0168] In one embodiment, the sorting unit 307 includes:

[0169] The reverse sorting subunit is used to read relevant records about metadata from the specified location and arrange them in reverse order to obtain sorted metadata records; the traversal copying subunit is used to traverse the sorted metadata records and copy the data files of the corresponding dates from the corresponding magnetic tapes to the destination in turn.

[0170] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned data lake table tape backup device 300 and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0171] The aforementioned data lake surface tape backup device 300 can be implemented as a computer program, which can, for example... Figure 7 It runs on the computer device shown.

[0172] Please see Figure 7 , Figure 7 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0173] See Figure 7 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0174] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a data lake table magnetic tape backup method.

[0175] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0176] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a data lake table tape backup method.

[0177] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0178] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps:

[0179] Scan the list of tables to be backed up to determine if the tables can be obtained. If the tables can be obtained, determine if there are tables to be backed up within them. If there are tables to be backed up, back them up according to their type. If the tables cannot be obtained, write all records since the last snapshot backup to a new Iceberg table for all tables involved in this backup, copy the relevant data to the specified tape directory, and insert the corresponding records into the metadata table.

[0180] In one embodiment, when the processor 502 performs the step of determining whether a table to be backed up exists in the table, it specifically implements the following steps:

[0181] For each table, the metadata table is queried to obtain the last backup time. If this is the first backup, the last backup time is set to zero. The most recent modification time of the table is obtained. It is determined whether the most recent modification time is not greater than the last backup time. If the most recent modification time is not greater than the last backup time, the scan of the list of tables to be backed up is performed to determine whether there are any tables that need to be backed up. If the most recent modification time is greater than the last backup time, the table is determined to be a table to be backed up.

[0182] In one embodiment, when implementing the backup step based on the type of the table to be backed up, the processor 502 specifically implements the following steps:

[0183] When the type is Base, copy the entire snapshot's data and metadata files and record them in the metadata table; when the type is Daily backup, only back up the data files added compared to the last snapshot and the updated metadata, and record them in the metadata table; when the type is Merge backup, merge multiple consecutive Daily backups into one Merge backup and update the metadata.

[0184] In one embodiment, when processor 502 implements the step of copying the entire snapshot's data and metadata file and recording it to the metadata table when the type is Base, the specific implementation steps are as follows:

[0185] When the type is Base, all data files, manifest files, manifest list files, and metadata files in the entire snapshot are copied to the specified tape directory, and relevant information is recorded in the metadata table.

[0186] In one embodiment, when the processor 502 implements the step of backing up only the data files newly added compared to the last snapshot and the updated metadata when the type is Daily backup, and recording them to the metadata table, the specific implementation steps are as follows:

[0187] When the type is Daily backup, compare the differences between the current snapshot and the last backup snapshot, back up only the newly added data files and their related manifest, manifest list and metadata files, and add new records to the metadata table.

[0188] In one embodiment, after performing the backup step based on the type of the table to be backed up, the processor 502 further performs the following steps:

[0189] When restoring the table, the metadata table is searched for the tape information where the backup for the target date is located; the specified tape is mounted according to the tape information, and the metadata for the target date is copied to the specified location; the metadata records are arranged in reverse order of the backup date, and the data files are copied from the corresponding tapes to the destination in sequence.

[0190] In one embodiment, when the processor 502 implements the step of arranging metadata records in reverse order of backup date and sequentially copying data files from the corresponding tape to the destination, the following steps are specifically implemented:

[0191] Read relevant records about metadata from the specified location and sort them in reverse order to obtain sorted metadata records; traverse the sorted metadata records and copy the data files of the corresponding dates from the corresponding tapes to the destination in turn.

[0192] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0193] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0194] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform the following steps:

[0195] Scan the list of tables to be backed up to determine if the tables can be obtained. If the tables can be obtained, determine if there are tables to be backed up within them. If there are tables to be backed up, back them up according to their type. If the tables cannot be obtained, write all records since the last snapshot backup to a new Iceberg table for all tables involved in this backup, copy the relevant data to the specified tape directory, and insert the corresponding records into the metadata table.

[0196] In one embodiment, when the processor executes the computer program to perform the step of determining whether a table to be backed up exists in the table, it specifically implements the following steps:

[0197] For each table, the metadata table is queried to obtain the last backup time. If this is the first backup, the last backup time is set to zero. The most recent modification time of the table is obtained. It is determined whether the most recent modification time is not greater than the last backup time. If the most recent modification time is not greater than the last backup time, the scan of the list of tables to be backed up is performed to determine whether there are any tables that need to be backed up. If the most recent modification time is greater than the last backup time, the table is determined to be a table to be backed up.

[0198] In one embodiment, when the processor executes the computer program to implement the backup step based on the type of the table to be backed up, it specifically implements the following steps:

[0199] When the type is Base, copy the entire snapshot's data and metadata files and record them in the metadata table; when the type is Daily backup, only back up the data files added compared to the last snapshot and the updated metadata, and record them in the metadata table; when the type is Merge backup, merge multiple consecutive Daily backups into one Merge backup and update the metadata.

[0200] In one embodiment, when the processor executes the computer program to implement the step of copying the entire snapshot's data and metadata file and recording it to the metadata table when the type is Base, the specific implementation is as follows:

[0201] When the type is Base, all data files, manifest files, manifest list files, and metadata files in the entire snapshot are copied to the specified tape directory, and relevant information is recorded in the metadata table.

[0202] In one embodiment, when the processor executes the computer program to implement the step of backing up only the data files newly added compared to the last snapshot and the updated metadata when the type is Daily backup, and recording them to the metadata table, the specific implementation is as follows:

[0203] When the type is Daily backup, compare the differences between the current snapshot and the last backup snapshot, back up only the newly added data files and their related manifest, manifest list and metadata files, and add new records to the metadata table.

[0204] In one embodiment, after executing the computer program to perform the backup step based on the type of the table to be backed up, the processor further performs the following steps:

[0205] When restoring the table, the metadata table is searched for the tape information where the backup for the target date is located; the specified tape is mounted according to the tape information, and the metadata for the target date is copied to the specified location; the metadata records are arranged in reverse order of the backup date, and the data files are copied from the corresponding tapes to the destination in sequence.

[0206] In one embodiment, when the processor executes the computer program to implement the step of arranging metadata records in reverse order of backup date and sequentially copying data files from the corresponding tape to the destination, it specifically implements the following steps:

[0207] Read relevant records about metadata from the specified location and sort them in reverse order to obtain sorted metadata records; traverse the sorted metadata records and copy the data files of the corresponding dates from the corresponding tapes to the destination in turn.

[0208] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0209] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0210] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0211] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0212] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0213] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data lake table tape backup method, characterized in that, include: Scan the list of tables to be backed up to determine if the tables can be retrieved; If the table can be obtained, determine whether there is a table to be backed up in the table; If there are tables to be backed up, back them up according to the type of the tables to be backed up; If the tables cannot be obtained, for all tables involved in this backup, all records since the last snapshot backup will be written into a new Iceberg table, the relevant data will be copied to the specified tape directory, and the corresponding records will be inserted into the metadata table.

2. The data lake table tape backup method according to claim 1, characterized in that, The step of determining whether a table to be backed up exists in the table includes: For table queries, the metadata table is used to obtain the last backup time. If the current backup is the first one, the last backup time is set to zero. Get the most recent modification time of the table; Determine whether the most recent modification time is not greater than the last backup time; If the most recent modification time is not greater than the last backup time, then the scan of the list of tables to be backed up is performed to determine whether there are any tables that need to be backed up. If the most recent modification time is greater than the last backup time, then the table is determined to be a table to be backed up.

3. The data lake table tape backup method according to claim 1, characterized in that, The step of backing up the table according to its type includes: When the type is Base, copy the entire snapshot's data and metadata file, and record it to the metadata table; When the type is Daily backup, only the data files newly added and the updated metadata compared to the last snapshot are backed up and recorded in the metadata table; When the type is Merge backup, multiple consecutive Daily backups are merged into one Merge backup, and the metadata is updated.

4. The data lake table tape backup method according to claim 3, characterized in that, When the type is Base, the entire snapshot's data and metadata file are copied and recorded in the metadata table, including: When the type is Base, all data files, manifest files, manifestlist files, and metadata files in the entire snapshot are copied to the specified tape directory, and relevant information is recorded in the metadata table.

5. The data lake table tape backup method according to claim 3, characterized in that, When the type is Daily backup, only the data files newly added and the updated metadata compared to the last snapshot are backed up and recorded in the metadata table, including: When the type is Daily backup, compare the differences between the current snapshot and the last backup snapshot, back up only the newly added data files and their related manifest, manifest list and metadata files, and add new records to the metadata table.

6. The data lake table tape backup method according to claim 1, characterized in that, After backing up the table according to its type, the process includes: When restoring the table, look up the tape information where the backup for the target date was located in the metadata table; Mount the specified tape according to the tape information, and copy the metadata of the target date to the specified location; Metadata records are sorted in reverse order of backup date, and data files are copied from the corresponding tapes to the destination in sequence.

7. The data lake table tape backup method according to claim 6, characterized in that, The metadata records are arranged in reverse order of backup date, and data files are copied sequentially from the corresponding tapes to the destination, including: Read relevant records about metadata from the specified location and arrange them in reverse order to obtain sorted metadata records; Iterate through the sorted metadata records and copy the data files corresponding to the dates from the corresponding tapes to the destination.

8. A data lake table magnetic tape backup device, characterized in that, include: The scanning unit is used to scan the list of tables to be backed up and determine whether the tables can be obtained. The judgment unit is used to determine whether a table to be backed up exists in the table if the table can be obtained. A backup unit is used to back up the table according to its type. The update unit is used to write all records since the last snapshot backup into a new Iceberg table for all tables involved in this backup if the table cannot be obtained, copy the relevant data to the specified tape directory, and insert the corresponding records into the metadata table.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images