Method, device, equipment, storage medium and program product for online migration of backup system virtual machine granularity history data

Abstract

This application relates to an online migration method, apparatus, device, storage medium, and program product for historical data at the virtual machine granularity of a backup system. By migrating historical backup data from a first data table to a second data table, if a newly generated backup result is detected to include a first format, it is converted to a second format. The backup results in the first and second formats are then stored in their respective formatted data tables. Compared to traditional offline data migration methods, this application migrates data from the first data table to the second data table online. During the migration, if a newly generated backup result is detected, and if it includes the first format, the backup result in the first format is converted to the second format, and both formats are stored. This achieves the migration of newly generated backup results within the online migration of historical backup data, improving the integrity of the backup data migration.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to an online migration method, apparatus, computer equipment, computer-readable storage medium, and computer program product for backing up historical data at the virtual machine granularity level. Background Technology

[0002] In a virtual machine backup system, after each backup job is completed, the system records the backup history data for each virtual machine in that job. As the number of virtualization platforms supported by the backup system increases, the granularity of historical data recording evolves from one record per backup job to one record per virtual machine. To adapt to the new format of backup history data storage, the old format backup history data needs to be migrated to the new format data structure. Currently, the method for migrating historical data in a backup system is usually to migrate the old format historical data to the new format data table structure offline after creating the new format data structure. However, historical data in the backup system is continuously generated, and migrating it offline will result in incomplete data and reduce the integrity of the historical data.

[0003] Therefore, current methods for migrating historical data at the virtual machine granularity of backup systems suffer from low integrity. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for online migration of virtual machine-level historical data in a backup system that can improve the integrity of the above-mentioned technical problems.

[0005] Firstly, this application provides an online migration method for historical data at the virtual machine granularity of a backup system, including:

[0006] When the conditions for migrating backup history data are met, the backup history data of the first data table is migrated to the second data table through a data format converter; the first data table adopts a first format, and the second data table adopts a second format; the first format represents the format for storing backup history data in units of backup jobs, and the second format represents the format for storing backup history data in units of virtual machines;

[0007] During the migration of historical backup data, if a newly generated backup result corresponding to the virtual machine is detected, the data format of the backup result is obtained;

[0008] If the data format does not include the second format but includes the first format, then the backup result in the first format is converted into a backup result in the second format by the data format converter, and the backup result in the first format and the backup result in the second format are stored in the corresponding format data tables respectively to obtain the backup history data corresponding to the backup result.

[0009] Secondly, this application also provides an online migration device for backing up historical data at the virtual machine granularity of a backup system, comprising:

[0010] The first migration module is used to migrate the backup history data of the first data table to the second data table through a data format converter when the conditions for backing up historical data migration are met. The first data table adopts a first format, and the second data table adopts a second format. The first format represents the format for storing backup history data in units of backup jobs, and the second format represents the format for storing backup history data in units of virtual machines.

[0011] The detection module is used to obtain the data format of the backup result if a newly generated backup result corresponding to the virtual machine is detected during the migration of historical backup data.

[0012] The second migration module is used to convert the backup result in the first format to the backup result in the second format through the data format converter if the data format does not include the second format but includes the first format, and to store the backup result in the first format and the backup result in the second format into data tables of corresponding formats respectively, so as to obtain the backup history data corresponding to the backup result.

[0013] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0014] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0015] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0016] The aforementioned online migration method, apparatus, computer equipment, computer-readable storage medium, and computer program product for virtual machine-level historical data of a backup system migrates backup historical data from a first data table in a first format to a second data table in a second format using a data format converter. During the migration process, if a newly generated backup result is detected, its data format is obtained. If it does not include the second format but includes the first format, the backup result in the first format is converted to the backup result in the second format. The backup results in the first and second formats are then stored in data tables of their respective formats to obtain the backup historical data. Compared to the traditional offline method of creating a new data structure in a new format and then migrating the data, this application utilizes a data format converter to migrate data from a first data table in the first format to a second data table in the second format online. When a newly generated backup result is detected during the migration, if it does not include the second format but includes the first format, the backup result in the first format is converted to the backup result in the second format, and both the backup results in the first and second formats are stored. This achieves the migration of newly generated backup results during online migration of backup historical data, improving the integrity of the backup data migration. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating an online migration method for backing up historical data at the virtual machine level in one embodiment.

[0019] Figure 2 This is a flowchart illustrating the migration steps in one embodiment;

[0020] Figure 3 This is a flowchart illustrating an online migration method for backing up historical data at the virtual machine granularity in another embodiment.

[0021] Figure 4 This is a structural block diagram of an online migration device for backing up historical data at the virtual machine granularity in one embodiment;

[0022] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0025] In related technologies, within virtual machine backup systems, after each backup job is completed, the system records the backup result (success / failure / skipped, backup time, data volume, etc.) for each virtual machine, forming backup history data. This history data is used for backup report display, job status retrospection, compliance auditing, and other scenarios. As the number of virtualization platforms supported by the backup system increases, the granularity of historical data recording has evolved from the job level (one record per backup job) to the virtual machine level (one record per virtual machine). This evolution in granularity necessitates changes to the data storage structure.

[0026] The old format stores virtual machine-level historical data in the `extended_attrs` field of the `job_history` table, embedded as a Lightweight Data Interchange (JDI) string (JavaScript Object Notation, JSON). JSON stores data as key-value pairs. Each job history entry's `extended_attrs` field may contain backup information for dozens or even hundreds of virtual machines, with all virtual machine data serialized into a single JSON string. The new format stores virtual machine-level historical data in a separate `job_history_vm_entry` table, one record per virtual machine, containing structured fields such as virtual machine identifier (`vm_id`), virtual machine name (`vm_name`), host name (`host_name`), start time (`start_time`), end time (`end_time`), source data size (`source_size`), and status (`status`)).

[0027] Migrating from the old format to the new format presents the following challenges: No downtime during migration: The backup system is a 24 / 7 production system; backup services cannot be stopped during historical data migration, otherwise the backup window will be missed, leading to data protection interruption. No data loss: Historical data is the basis for compliance audits and backup reports; the loss of any record during migration will result in data inconsistency. Huge data volume: Long-running backup systems may accumulate millions of job history records, each containing information from multiple virtual machines, with the total data volume potentially reaching several gigabytes, requiring an efficient batch migration mechanism. Significant format differences: The old format's time field may be an ISO8601 string, a numeric timestamp, or a placeholder; the status field is represented by a combination of multiple fields (success, state, retry). The new format uses a unified bitmask (a technique that uses binary bits to represent multiple states) to represent status. For example, lock-free migration can be achieved by creating new tables, batch copying data, and establishing triggers to synchronize incremental changes. However, general-purpose database migration tools are unaware of the business semantics of the backup system (such as field mapping rules for different virtualization platforms, status bitmask conversion, and time format standardization), and cannot be directly applied to the migration of historical backup data. Triggers represent automatically executed programs on database tables that automatically trigger predefined operations when data changes.

[0028] This leads to the following problems: The virtual machine-level historical data of the backup system needs to be migrated from a JSON embedded format to an independent database table structure. The backup service cannot be interrupted during the migration, and new backup records need to be written to both the old and new storage systems simultaneously, lacking a unified dual-write consistency management mechanism. In the old format, different virtualization platforms use different field naming and value formats (e.g., time fields have three formats: ISO8601 string, numeric timestamp, and placeholder), lacking a universal multi-platform format parsing and standardized conversion method. Migrating massive amounts of historical data requires an efficient batch processing mechanism, while also supporting breakpoint resume (continuing from the last completed position after a migration interruption) to avoid starting the migration from scratch every time a restart occurs.

[0029] Based on this, this application utilizes a data format converter to migrate data from a first data table in a first format to a second data table in a second format online. When a newly generated backup result is detected during the migration, if it does not include the second format but includes the first format, the backup result in the first format is converted to the backup result in the second format. At the same time, the backup result in the first format and the backup result in the second format are stored. This enables the migration of newly generated backup results from historical backup data online, thereby improving the integrity of backup data migration.

[0030] This application employs a dual-write consistency mechanism to simultaneously write to both the old and new storage systems during migration, ensuring that newly generated backup data is not lost; it supports field mapping and format standardization conversion for various virtualization platforms (including time format normalization, status bitmask synthesis, and platform-specific field extraction) through a multi-platform legacy format parser; it efficiently processes massive amounts of historical data through a cursor-based breakpoint resume batch migration mechanism, supporting index pre-deletion and batch commit optimization; and it ensures the safety of repeated migration execution through idempotent writes.

[0031] In one embodiment, such as Figure 1 As shown, an online migration method for historical data at the virtual machine granularity of a backup system is provided. This embodiment illustrates the method by applying it to a server. It is understood that this method can also be applied to a terminal, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server, including the following steps S202 to S206. Wherein:

[0032] Step S202: When the conditions for backing up historical data migration are met, the backup historical data of the first data table is migrated to the second data table through a data format converter; the first data table adopts a first format and the second data table adopts a second format; the first format represents the format for storing backup historical data in units of backup jobs and the second format represents the format for storing backup historical data in units of virtual machines.

[0033] In this context, data recording and management are performed on a per-virtual-machine basis, unlike backups performed on the basis of an entire backup job. Backup history data can be generated based on the backup results of the corresponding virtual machines. The server can be a device used for migrating backup history data. The server can connect to both a first data table and a second data table. The first data table stores backup history data using a first format, and the second data table stores backup history data using a second format. The first format represents the format for storing backup history data on a per-backup-job basis; for example, after each backup job, backup history data for each virtual machine is generated and integrated into the backup history data for the current backup job. This first format can also be called the old format. The second format represents the format for storing backup history data on a per-virtual-machine basis; for example, a separate data sequence is set up for each virtual machine to store the backup history data for each backup. This second format can also be called the new format. To upgrade the storage data structure of backup history data, the server can migrate backup history data from the first data table to the second data table.

[0034] The conditions for migrating backup historical data can be triggered in various ways. For example, the conditions for migrating backup historical data include, but are not limited to, triggering and responding to backup historical data migration instructions at preset intervals. A data format converter can be used to convert backup historical data in a first format into backup historical data in a second format, so that the backup historical data in the second format can be stored in a second data table, thereby realizing the migration of backup historical data.

[0035] Step S204: In the migration of historical backup data, if a newly generated backup result corresponding to the virtual machine is detected, the data format of the above backup result is obtained.

[0036] The server can migrate historical backup data when the conditions for backing up historical data are met. During the migration of historical backup data, the backup process for virtual machines can continue, allowing the server to detect whether new backup results for virtual machines exist. If the server detects a newly generated backup result for a virtual machine during the historical backup data migration process, it obtains the data format of the backup result. The data format of the backup result refers to the storage format of the data in the backup result, including a first format and a second format.

[0037] Step S206: If the data format does not include the second format but includes the first format, the backup result in the first format is converted into the backup result in the second format by the data format converter, and the backup result in the first format and the backup result in the second format are stored in the corresponding format data tables respectively to obtain the backup history data corresponding to the backup result.

[0038] The server determines the storage strategy for the backup results based on the format contained in the newly generated backup results. Specifically, if the server detects that the data format in the backup results includes a second format, the server directly stores the backup results in a second data table, i.e., stores the backup results in the second format, thus obtaining backup history data in the second format.

[0039] If the server detects that the data format does not include the second format but includes the first format, the server can use the data format converter to convert the backup result in the first format to the backup result in the second format, and store the backup result in the first format and the backup result in the second format into data tables of the corresponding formats, respectively, to obtain the backup history data corresponding to the backup result. For example, the server stores the backup result in the first format into the first data table and the converted backup result in the second format into the second data table, thereby avoiding anomalies when accessing backup history data in the old format and improving data access compatibility.

[0040] In the aforementioned online migration method for virtual machine-level historical data of the backup system, a data format converter is used to migrate the backup historical data of a first data table using a first format to a second data table using a second format. During the migration process, if a newly generated backup result is detected, the data format of the backup result is obtained. If it does not include the second format but includes the first format, the backup result in the first format is converted to the backup result in the second format. The backup results in the first and second formats are then stored in the corresponding format data tables to obtain the backup historical data. Compared to the traditional offline method of creating a new data structure in a new format and then migrating the data, this application uses a data format converter to migrate the data in the first data table in the first format to the second data table in the second format online. When a newly generated backup result is detected during the migration, if it does not include the second format but includes the first format, the backup result in the first format is converted to the backup result in the second format, and both the backup results in the first and second formats are stored. This achieves the migration of newly generated backup results during the online migration of backup historical data, improving the integrity of the backup data migration.

[0041] In one embodiment, migrating backup history data from a first data table to a second data table via a data format converter includes: extracting backup history data entries corresponding to each virtual machine from the backup history data in the first data table using the data format converter; converting the backup history data entries corresponding to each virtual machine from a first format to a second format to obtain backup history data entries in the second format; and storing the backup history data entries in the second format in the second data table.

[0042] In this embodiment, the backup history data in the first data table can be stored in units of a single backup job. To migrate the backup history data in the first data table to a new format, the server can use a data format converter to extract the backup history data entries corresponding to each virtual machine from the backup history data in the first data table. For example, the server queries the backup history data entries of the virtual machines corresponding to the identification information from each backup job based on the identification information of the virtual machines, and obtains the backup history data entries corresponding to the virtual machines by integrating the various backup history data entries. For each backup history data entry corresponding to the virtual machine, the server can convert the backup history data entry from the first format to the second format to obtain the backup history data entry in the second format. For example, the server performs format conversion through field mapping, format changes, etc., so that the server can store the backup history data entry in the second format in the second data table, realizing the migration of backup history data.

[0043] Specifically, such as Figure 2 As shown, Figure 2 This is a flowchart illustrating the migration steps in one embodiment. For the backup history data in the first data table, the server first deletes the index of the target table (such as the first data table) to optimize write performance. Then, it queries the identifier (MAX job_history_id) of the largest backup history data in the target table as a migration breakpoint. Subsequently, it reads 5,000 old records based on this breakpoint. After that, it checks whether the records read are empty. If the records are empty, it directly rebuilds the target table index and ends the overall migration process. If there are valid records, it parses the extended attributes (extended_attrs) and converts them into virtual machine (VM) entries (second format). Then, it writes the processed data into a new table (second data table) by ignoring duplicate insert statements (INSERT IGNORE). After writing is complete, it checks whether the cumulative processed data has reached 50,000 records. If it has not reached 50,000 records, it returns to the step of reading 5,000 old records to continue the loop processing. If the cumulative total has reached 50,000 records, it first executes a transaction commit. After the commit is completed, it also returns to the step of reading 5,000 old records to continue processing the remaining data. Here, the extended attributes (extended_attrs) represent the JSON fields that store virtual machine granular history data in the old format. A transaction represents an atomic unit of database operation; it either succeeds entirely or rolls back entirely. INSERT IGNORE is an SQL statement that skips the insertion process instead of raising an error when a primary key conflict is encountered, ensuring idempotency.

[0044] The batch migration of historical data (backup historical data in the first data table) is performed through the following steps:

[0045] Step 1: Index Pre-deletion. Before migration, delete the indexes (data structures in the database that speed up queries) of the target table. This is because each record in a batch insert requires updating the index, and the presence of an index will significantly slow down the insertion process. Indexes represent data structures in the database used to accelerate queries, but they reduce the speed of batch inserts.

[0046] Step 2: Breakpoint Location. Query the maximum existing `job_history_id` in the new table (the second data table) and continue migration from that position to avoid duplicate processing: `SELECT MAX(job_history_id) FROM job_history_vm_entry`. This statement means querying the maximum value (MAX) of the identifier (`job_history_id`) of the backup historical virtual machine entry table (`job_history_vm_entry`) (the second data table).

[0047] Step 3: Batch Reading and Conversion. The server reads data from the old table (first data table) in ascending order by job_history_id, 5000 records per batch: SELECT * FROM job_history WHERE job_history_id>:last_id ORDERBY job_history_id LIMIT 5000. This statement means: query all fields (*) from the historical backup table (job_history) (first data table), filter out data whose backup history data identifier (job_history_id) is greater than the last number of the previous batch (:last_id), sort them in ascending order by backup history data identifier (job_history_id), and read a maximum of 5000 records (LIMIT 5000). Each record is parsed using a legacy format converter (data format converter) to extract the virtual machine entry (the backup history data entry corresponding to the virtual machine).

[0048] Step 4: Batch Write. Write the transformed virtual machine entries to a new table using the INSERT IGNORE method described above. Commit a transaction every 50,000 records.

[0049] Step 5: Index Rebuilding. After the migration is complete, rebuild the indexes of the target table to restore query performance.

[0050] The server also implements a resume guarantee: if the system restarts or is abnormally interrupted during the migration process, it will resume from the MAX(job_history_id) position on the next startup, and migrated records will be automatically skipped due to INSERT IGNORE. The server also implements an idempotency guarantee: multiple migrations of the same record will not produce duplicate data because INSERT IGNORE silently skips records when it encounters a primary key conflict. Idempotency means that performing the same operation multiple times yields the same result as performing it once, without producing side effects.

[0051] In this embodiment, the server can use a data format converter to extract the backup history data entries corresponding to each virtual machine from the first data table and perform format conversion, thereby improving the efficiency of backup history data migration.

[0052] In one embodiment, converting the backup history data entry from a first format to a second format to obtain a backup history data entry in the second format includes: obtaining attribute information, time information, and status information from the backup history data entry; and converting the attribute information, time information, and status information from the first format to the second format respectively to obtain a backup history data entry in the second format.

[0053] In this embodiment, the extracted backup history data entries corresponding to the virtual machines include various types of information. For example, they may include attribute information, time information, and status information. Attribute information represents the attributes of the virtual machine, time information represents the time information in the backup history data, and status information represents the backup status information in the backup history data. The server can obtain the attribute information, time information, and status information from the aforementioned backup history data entries, and convert these information from the first format to the second format to obtain backup history data entries in the second format. The attribute information includes, but is not limited to, virtual machine identifier, virtual machine name, host identifier, host name, cluster identifier, IP address, and project information.

[0054] Through this embodiment, the server can use a data format converter to convert the format of various information in the backup historical data, thereby improving the efficiency of backing up historical data migration.

[0055] In one embodiment, converting the attribute information, time information, and status information from the first format to the second format to obtain backup historical data entries in the second format includes: for each attribute information, mapping the attribute information in the first format to target attribute information in the second format; the attribute type represented by the target attribute information is the same as the attribute type represented by the attribute information; for each time information, converting the time information in the first format to target time information in the second format according to the type of the time information using a corresponding strategy; for each status information, obtaining each status bitmask corresponding to the status information, and converting each status bitmask in the first format to a target status bitmask in the second format; the target status bitmask represents the same target status information as the status information; and obtaining backup historical data entries in the second format based on the target attribute information, the target time information, and the target status bitmask.

[0056] In this embodiment, for different types of information in the backup historical data entries, the server can perform format conversion using different methods through a data format converter. Specifically, for each attribute information, the server can perform format conversion using field mapping, mapping the attribute information in the first format to the target attribute information in the second format. The attribute type represented by the target attribute information is the same as the attribute type represented by the attribute information itself. That is, the server converts the format without changing the meaning of the attribute information fields. For each time information, the server can convert the time information in the first format to the target time information in the second format using a strategy corresponding to the time information type. That is, the server uses different strategies for format conversion for different types of time information. For each status information, the server can obtain the corresponding status bitmasks, where the status information can be composed of one or more status bitmasks. The server converts the status bitmasks in the first format to the target status bitmasks in the second format. Here, a bitmask represents a technique that uses combinations of binary bits to represent multiple states, with each bit representing a state switch. The target state bitmask described above represents the same target state information as the aforementioned state information; that is, the server changes the data format but does not change the meaning of the state information. Therefore, the server can obtain backup historical data entries in the second format based on the aforementioned target attribute information, target time information, and target state bitmask.

[0057] Specifically, the data format converter, also known as a legacy format converter, supports format conversion across multiple virtual machine platforms. It supports parsing historical data (backup historical data) from various virtualization platforms, and its core functions include:

[0058] Field mapping: Different platforms use different field names to represent the same information. Data format converters handle all platforms through a unified parse_vm_entry() function (virtual machine table parsing function), for example, through field mapping. Specifically, this can be represented as follows: For virtual machine identifiers, the old format (first format) field is vm_id (virtual machine identifier), and the new format (second format) field is vm_id (virtual machine identifier); for virtual machine names, the old format fields are title and name (virtual machine name), and the new format field is vm_display_name (virtual machine display name); for host identifiers, the old format fields are host and host_id (host identifier), and the new format field is host_id (host identifier); for host names, the old format field is host (host name), and the new format field is host_name (host name); for cluster identifiers, the old format fields are cluster and cluster_id (cluster identifier), and the new format field is cluster_id (cluster identifier); for IP addresses, the old format field is ips (comma-separated IP string), and the new format field is ip_addrs (set of IP addresses); for project information, the old format fields are project and tenant (project information), and the new format fields are project_id (project number) and project_name (project name).

[0059] The server can also standardize the time format using a data format converter. In the old format, the time field had three representations, which the converter uniformly converts to the new DATETIME type. Specifically: for old ISO 8601 string formats, such as "20xx-x-xxT00:00:00Z", the processing method is direct parsing and use; for old digital timestamp formats, such as 1x04067200, the processing method is conversion to standard date and time; for old placeholder formats, such as "-", the processing method is assignment of the null value NULL. ISO 8601 is an internationally standardized date and time representation format.

[0060] The server can also synthesize status bitmasks using a data format converter. In the old format, the virtual machine backup status was scattered across multiple fields (succeed boolean value, state string, and retry flag). The converter can combine these fields into a bitmask status in the new format. Specifically, when the old format content is succeed=true (execution successful = true) and state="completed" (status = completed), the corresponding new format status is COMPLETED (completed), and the corresponding bitmask value is bit 0; when the old format content is succeed=false (execution successful = false) and state="failed" (status = failed), the corresponding new format status is FAILED (failed), and the corresponding bitmask value is bit 1; when the old format content is state="skipped" (status = skipped), the corresponding new format status is SKIPPED (skipped), and the corresponding bitmask value is bit 2; when the old format content is retry=true (retry = true), the corresponding new format status is RETRY (retry), and the corresponding bitmask value is bit 3.

[0061] Through this embodiment, the server can perform format conversion for different types of backup historical data using corresponding methods, thereby improving the efficiency of migrating backup historical data.

[0062] In one embodiment, after obtaining the data format of the backup result, the method further includes: if the data format includes the second format, then writing the backup result of the second format into the second data table to obtain the backup history data corresponding to the backup result.

[0063] In this embodiment, the server employs different storage methods for backup results containing different formats. If the server detects that a newly generated backup result includes a second format (i.e., a new format backup result), the server can write the second format backup result into the second data table to obtain the corresponding backup history data. For backup results containing the first format, the server can use a dual-write consistency mechanism to store the converted second format backup result and the first format backup result into the second data table and the first data table respectively, ensuring that the reader can access the corresponding backup history data using both the old and new storage.

[0064] Specifically, dual-write refers to simultaneously writing data to both the old and new data stores during migration to ensure data consistency. The idea behind the dual-write consistency mechanism is that during migration, each newly generated backup data (backup result) is written to both the old and new storage systems simultaneously, ensuring that the reader can obtain complete data regardless of which storage system is used. The write process can be represented as follows: After the backup job is completed, the Job History Manager receives the virtual machine's backup result; the server first checks whether the result contains vm_entries (a structured list of virtual machine entries) in the new format (second format); if it contains new format data, the server directly writes it to the new table (second data table) job_history_vm_entry; if it does not contain new format data, the server checks whether it contains extended_attrs (JSON strings) in the old format (first format); if it contains old format data, the server parses and converts it to the new format using a data format converter (Legacy_VM_History_Converter) and writes it to the new table; simultaneously, the server keeps the old format data written to the old storage (first data table) to ensure compatibility for readers of older versions.

[0065] To ensure consistency, the server writes to the new table and the old storage within the same transaction (an atomic unit of database operations). The server uses the INSERT IGNORE statement to write to the new table, ensuring that repeated executions do not produce duplicate records (idempotency). Furthermore, the server prioritizes using data from the new table when reading data; if the new table has no data, it falls back to parsing the old format.

[0066] Through this embodiment, the server can use different methods to store backup results in different formats, thereby improving the integrity of backup history data.

[0067] In one embodiment, the method further includes: if a breakpoint resume signal for the backup history data is detected, obtaining the identifiers corresponding to each of the migrated backup history data from the second data table; the identifiers represent the storage time of the backup history data in the first data table; the numerical value of the identifiers corresponding to the backup history data is positively correlated with the storage time; determining the target backup history data corresponding to the identifier with the largest numerical value; and migrating the backup history data from the first data table to the second data table using the data format converter, starting from the target backup history data and following the numerical order of the identifiers.

[0068] In this embodiment, resuming interrupted downloads means continuing processing from the point of last interruption, avoiding duplication of completed work. When storing backup historical data to the data table, the server can add a corresponding identifier to the backup historical data based on its storage time. This identifier represents the storage time of the backup historical data in the first data table, and the value of the identifier corresponding to the backup historical data is positively correlated with the storage time. For example, the earlier the storage time, the smaller the identifier value, and the later the storage time, the larger the identifier value.

[0069] If an interruption occurs during the migration of the aforementioned backup historical data, the server can resume the transmission after detecting a breakpoint resume signal for the backup historical data. Specifically, the server can retrieve the identifiers corresponding to each migrated backup historical data from the second data table. The server then determines the target backup historical data corresponding to the identifier with the largest value, and using this target backup historical data as a starting point, migrates the backup historical data from the first data table to the second data table in numerical order of the identifiers, using the data format converter.

[0070] Specifically, during breakpoint location, the server queries the new table (the second data table) for the maximum existing `job_history_id` (the identifier of the backup history data), and continues the migration from that position to avoid duplicate processing: `SELECT MAX(job_history_id) FROM job_history_vm_entry`. This statement means querying the maximum value (MAX) of the identifier (`job_history_id`) of the backup history data from the historical backup virtual machine entry table (`job_history_vm_entry`) (the second data table).

[0071] In this embodiment, the server supports efficient migration of massive amounts of historical data through a breakpoint-resume batch migration mechanism. It processes multiple data entries in parallel, commits transactions at a frequency of multiple entries per transaction, and optimizes with index pre-deletion, resulting in migration speeds tens of times faster than processing data entry by entry. Migration can resume from the breakpoint after an interruption, eliminating the need to start from scratch and further improving data migration efficiency.

[0072] In one exemplary embodiment, such as Figure 3 As shown, Figure 3 This is a flowchart illustrating an online migration method for backing up historical data at the virtual machine granularity level in another embodiment. This embodiment includes the following:

[0073] The server comprises a backup service layer, a dual-write management layer, a data storage layer, and a batch migration layer. The backup job executor in the backup service layer initiates a call to the job history manager (Job_History_Manager) in the dual-write management layer. It then checks if new-format virtual machine entries (vm_entries) exist in the data. If they do, the data is directly written to the new storage backup history data virtual machine entry table (job_history_vm_entry) in the data storage layer. If not, it checks if the old-format extended attribute field (extended_attrs) is present. If the old-format extended attribute field (extended_attrs) does not exist, only the extended attribute JSON field (extended_...) from the old storage backup history data table (job_history) in the data storage layer is retained. The `attrs` function does not perform new table write operations. If an old format extended attribute field (`extended_attrs`) exists, it calls the data format converter (`Legacy_VM_History_Converter`) to complete the format conversion. After conversion, a compatible write operation is performed, synchronously writing the extended attribute JSON field (`extended_attrs`) of the old storage backup history data table (`job_history`) and the structured field of the new storage backup history data virtual machine entry table (`job_history_vm_entry`). In addition, a separate batch migration layer reads data from the old and new storage through a batch migrater with 5,000 records per batch. It uses a breakpoint cursor to obtain the backup history data with the largest identifier (`MAX job_history_id`) as the migration read breakpoint. At the same time, it relies on the index manager to perform a complete set of operations to delete indexes, migrate data, and rebuild indexes. The data source read by the batch migrater is the old storage backup history data table (`job_history`) in the data storage layer.

[0074] Specifically, regarding the dual-write consistency mechanism: The idea behind the dual-write consistency mechanism is that during migration, each newly generated backup data (backup result) is simultaneously written to both the old and new storage systems, ensuring that the reader can obtain complete data regardless of which storage system is used. The write process can be represented as follows: After the backup job is completed, the Job History Manager receives the virtual machine's backup result; the server first checks whether the result contains vm_entries (a structured list of virtual machine entries) in the new format (second format); if it contains new format data, the server directly writes it to the new table (second data table) job_history_vm_entry; if it does not contain new format data, the server checks whether it contains extended_attrs (JSON strings) in the old format (first format); if it contains old format data, the server parses and converts it to the new format using a data format converter (Legacy_VM_History_Converter) and writes it to the new table; simultaneously, the server keeps the old format data written to the old storage (first data table) to ensure compatibility for readers of older versions.

[0075] To ensure consistency, the server writes to the new table and the old storage within the same transaction (an atomic unit of database operations). The server uses the INSERT IGNORE statement to write to the new table, ensuring that repeated executions do not produce duplicate records (idempotency). Furthermore, the server prioritizes using data from the new table when reading data; if the new table has no data, it falls back to parsing the old format.

[0076] This is a multi-platform data format converter. Also known as a legacy format converter, it supports format conversion across multiple virtual machine platforms. The legacy format converter supports parsing historical data (backup historical data) from various virtualization platforms. Its core functions include:

[0077] Field mapping: Different platforms use different field names to represent the same information. Data format converters handle all platforms through a unified parse_vm_entry() function (virtual machine table parsing function), for example, through field mapping. Specifically, this can be represented as follows: For virtual machine identifiers, the old format (first format) field is vm_id (virtual machine identifier), and the new format (second format) field is vm_id (virtual machine identifier); for virtual machine names, the old format fields are title and name (virtual machine name), and the new format field is vm_display_name (virtual machine display name); for host identifiers, the old format fields are host and host_id (host identifier), and the new format field is host_id (host identifier); for host names, the old format field is host (host name), and the new format field is host_name (host name); for cluster identifiers, the old format fields are cluster and cluster_id (cluster identifier), and the new format field is cluster_id (cluster identifier); for IP addresses, the old format field is ips (comma-separated IP string), and the new format field is ip_addrs (set of IP addresses); for project information, the old format fields are project and tenant (project information), and the new format fields are project_id (project number) and project_name (project name).

[0078] The server can also standardize the time format using a data format converter. In the old format, the time field had three representations, which the converter uniformly converts to the new DATETIME (date and time) type. Specifically: for old ISO8601 string formats, such as "20xx-x-xxT00:00:00Z", the processing method is direct parsing and use; for old digital timestamp formats, such as 1x04067200, the processing method is conversion to standard date and time; for old placeholder formats, such as "-", the processing method is assignment of the null value NULL.

[0079] The server can also synthesize status bitmasks using a data format converter. In the old format, the virtual machine backup status was scattered across multiple fields (succeed boolean value, state string, and retry flag). The converter can combine these fields into a bitmask status in the new format. Specifically, when the old format content is succeed=true (execution successful = true) and state="completed" (status = completed), the corresponding new format status is COMPLETED (completed), and the corresponding bitmask value is bit 0; when the old format content is succeed=false (execution successful = false) and state="failed" (status = failed), the corresponding new format status is FAILED (failed), and the corresponding bitmask value is bit 1; when the old format content is state="skipped" (status = skipped), the corresponding new format status is SKIPPED (skipped), and the corresponding bitmask value is bit 2; when the old format content is retry=true (retry = true), the corresponding new format status is RETRY (retry), and the corresponding bitmask value is bit 3.

[0080] The server can perform a batch migration mechanism with breakpoint resume capability.

[0081] Batch migration of historical data (backup historical data in the first data table) is performed through the following steps:

[0082] Step 1: Index Pre-deletion. Before migration, delete the indexes (indexes in the database to speed up queries) of the target table, because each record needs to be updated during batch insertion, and the existence of indexes will seriously slow down the insertion speed.

[0083] Step 2: Breakpoint Location. Query the maximum existing `job_history_id` in the new table (the second data table) and continue migration from that position to avoid duplicate processing: `SELECT MAX(job_history_id) FROM job_history_vm_entry`. This statement means querying the maximum value (MAX) of the identifier (`job_history_id`) of the backup historical virtual machine entry table (`job_history_vm_entry`) (the second data table).

[0084] Step 3: Batch Reading and Conversion. The server reads data from the old table (first data table) in ascending order by job_history_id, 5000 records per batch: SELECT * FROM job_history WHERE job_history_id>:last_id ORDERBY job_history_id LIMIT 5000. This statement means: query all fields (*) from the historical backup table (job_history) (first data table), filter out data whose backup history data identifier (job_history_id) is greater than the last number of the previous batch (:last_id), sort them in ascending order by backup history data identifier (job_history_id), and read a maximum of 5000 records (LIMIT 5000). Each record is parsed using a legacy format converter (data format converter) to extract the virtual machine entry (the backup history data entry corresponding to the virtual machine).

[0085] Step 4: Batch Write. Write the transformed virtual machine entries to a new table using the INSERT IGNORE method described above. Commit a transaction every 50,000 records.

[0086] Step 5: Index Rebuilding. After the migration is complete, rebuild the indexes of the target table to restore query performance.

[0087] The server also implements a resume guarantee: if the system restarts or is abnormally interrupted during the migration process, it will resume from the MAX(job_history_id) position on the next startup, and the migrated records will be automatically skipped due to INSERT IGNORE. The server also implements an idempotency guarantee: multiple migrations of the same record will not generate duplicate data, because INSERT IGNORE will silently skip when it encounters a primary key conflict.

[0088] The server can deploy the data format converter to multiple virtualization platforms. The field mapping rules for each platform are distinguished by a platform identifier, and the converter automatically selects the corresponding mapping rule based on the platform identifier in extended_attrs.

[0089] Through the above embodiments, the server migrates data from a first data table in a first format to a second data table in a second format online using a data format converter. When a newly generated backup result is detected during the migration, if it does not include the second format but includes the first format, the backup result in the first format is converted to the backup result in the second format. At the same time, the backup result in the first format and the backup result in the second format are stored. This enables the migration of newly generated backup results from historical backup data online, thereby improving the integrity of backup data migration.

[0090] Furthermore, a dual-write consistency mechanism enables seamless migration of backup historical data from the old to the new format. During the migration, the backup service continues to run, and new backup data is written to both the old and new storage systems simultaneously. Readers can retrieve complete data from either storage system, preventing data loss. A multi-platform legacy format parser unifies the differences in historical data formats across various virtualization platforms, converting unstructured JSON strings into structured database records, providing a high-quality data foundation for new backup reports, compliance audits, and data analysis. Idempotent writes ensure the migration process can be safely and repeatedly executed. In the event of migration anomalies, operations personnel can easily re-trigger the migration; completed records are automatically skipped, eliminating the need for manual deduplication.

[0091] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0092] Based on the same inventive concept, this application also provides an online migration device for granular historical data of backup system virtual machines to implement the online migration method for granular historical data of backup system virtual machines as described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the online migration device for granular historical data of backup system virtual machines provided below can be found in the limitations of the online migration method for granular historical data of backup system virtual machines described above, and will not be repeated here.

[0093] In one exemplary embodiment, such as Figure 4 As shown, an online migration device for backing up historical data at the virtual machine granularity of a backup system is provided, comprising: a first migration module 500, a detection module 502, and a second migration module 504, wherein:

[0094] The first migration module 500 is used to migrate the backup history data of the first data table to the second data table through a data format converter when the conditions for backup history data migration are met. The first data table adopts a first format, and the second data table adopts a second format. The first format represents the format for storing backup history data in units of backup jobs, and the second format represents the format for storing backup history data in units of virtual machines.

[0095] The detection module 502 is used to obtain the data format of the backup result if a newly generated backup result corresponding to the virtual machine is detected during the migration of historical backup data.

[0096] The second migration module 504 is used to convert the backup result in the first format into the backup result in the second format by means of the data format converter if the data format does not include the second format but includes the first format, and to store the backup result in the first format and the backup result in the second format into the corresponding format data table respectively, so as to obtain the backup history data corresponding to the backup result.

[0097] In one embodiment, the first migration module 500 is configured to extract backup history data entries corresponding to each virtual machine from the backup history data of the first data table through the data format converter; for each backup history data entry corresponding to the virtual machine, convert the backup history data entry from the first format to the second format to obtain a backup history data entry in the second format; and store the backup history data entry in the second format in the second data table.

[0098] In one embodiment, the first migration module 500 is used to obtain attribute information, time information and status information from the backup history data entry; and convert the attribute information, time information and status information from the first format to the second format to obtain the backup history data entry in the second format.

[0099] In one embodiment, the first migration module 500 is configured to: map the attribute information in the first format to target attribute information in the second format for each of the attribute information; the attribute type represented by the target attribute information is the same as the attribute type represented by the attribute information; for each of the time information, convert the time information in the first format to target time information in the second format according to the type of the time information; for each of the state information, obtain each state bitmask corresponding to the state information, and convert each of the state bitmasks in the first format to target state bitmasks in the second format; the target state bitmask represents the same target state information as the state information; and obtain backup historical data entries in the second format based on the target attribute information, the target time information, and the target state bitmask.

[0100] In one embodiment, the apparatus further includes a storage module, configured to write the backup result of the second format into the second data table if the data format includes the second format, thereby obtaining the backup history data corresponding to the backup result.

[0101] In one embodiment, the apparatus further includes: a resume module, configured to, if a resume signal for the backup historical data is detected, obtain the identifiers corresponding to each migrated backup historical data from the second data table; the identifiers represent the storage time of the backup historical data in the first data table; the numerical value of the identifiers corresponding to the backup historical data is positively correlated with the storage time; determine the target backup historical data corresponding to the identifier with the largest numerical value; and, starting from the target backup historical data, migrate the backup historical data from the first data table to the second data table according to the numerical order of the identifiers through the data format converter.

[0102] The modules in the online migration device for virtual machine-level historical data of the aforementioned backup system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0103] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores backup historical data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an online migration method for backup system virtual machine-level historical data.

[0104] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0105] In one exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described online migration method for backup system virtual machine granular historical data.

[0106] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the online migration method for historical data at the virtual machine granularity of the backup system described above.

[0107] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method for online migration of historical data at the virtual machine granularity of a backup system.

[0108] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0109] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0110] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0111] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An online migration method for historical data at the virtual machine granularity of a backup system, characterized in that, The method includes: When the conditions for migrating backup history data are met, the backup history data of the first data table is migrated to the second data table through a data format converter; the first data table adopts a first format, and the second data table adopts a second format; the first format represents the format for storing backup history data in units of backup jobs, and the second format represents the format for storing backup history data in units of virtual machines; During the migration of historical backup data, if a newly generated backup result corresponding to the virtual machine is detected, the data format of the backup result is obtained; If the data format does not include the second format but includes the first format, then the backup result in the first format is converted into a backup result in the second format by the data format converter, and the backup result in the first format and the backup result in the second format are stored in the corresponding format data tables respectively to obtain the backup history data corresponding to the backup result.

2. The method according to claim 1, characterized in that, The process of migrating backup historical data from the first data table to the second data table using a data format converter includes: The backup history data entries corresponding to each virtual machine are extracted from the backup history data of the first data table using the data format converter. For each backup history data entry corresponding to the virtual machine, the backup history data entry is converted from the first format to the second format to obtain the backup history data entry in the second format. Store the backup history data entries in the second format into the second data table.

3. The method according to claim 2, characterized in that, The step of converting the backup history data entries from the first format to the second format to obtain backup history data entries in the second format includes: Obtain the attribute information, time information, and status information from the backup historical data entries; The attribute information, time information, and status information are converted from the first format to the second format to obtain backup historical data entries in the second format.

4. The method according to claim 3, characterized in that, The step of converting the attribute information, the time information, and the status information from the first format to the second format to obtain backup historical data entries in the second format includes: For each of the attribute information, the attribute information in the first format is mapped to target attribute information in the second format; the attribute type represented by the target attribute information is the same as the attribute type represented by the attribute information. For each piece of time information, according to the type of time information, a corresponding strategy is used to convert the time information in the first format into the target time information in the second format; For each state information, obtain each state bitmask corresponding to the state information, and convert each state bitmask in the first format into a target state bitmask in the second format; the target state bitmask represents the same target state information as the state information. Based on the target attribute information, the target time information, and the target status bitmask, backup historical data entries in a second format are obtained.

5. The method according to any one of claims 1, characterized in that, After obtaining the data format of the backup result, the method further includes: If the data format includes the second format, then the backup result of the second format is written into the second data table to obtain the backup history data corresponding to the backup result.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: If a breakpoint resume signal for the backup history data is detected, the identifier corresponding to each migrated backup history data is obtained from the second data table; the identifier represents the storage time of the backup history data in the first data table; the value of the identifier corresponding to the backup history data is positively correlated with the storage time. Determine the target backup historical data corresponding to the identifier with the largest value; Starting with the target backup historical data, the backup historical data of the first data table is migrated to the second data table according to the numerical order of the identifiers through the data format converter.

7. An online migration device for historical data at the virtual machine granularity of a backup system, characterized in that, The device includes: The first migration module is used to migrate the backup history data of the first data table to the second data table through a data format converter when the conditions for backing up historical data migration are met. The first data table adopts a first format, and the second data table adopts a second format. The first format represents the format for storing backup history data in units of backup jobs, and the second format represents the format for storing backup history data in units of virtual machines. The detection module is used to obtain the data format of the backup result if a newly generated backup result corresponding to the virtual machine is detected during the migration of historical backup data. The second migration module is used to convert the backup result in the first format to the backup result in the second format through the data format converter if the data format does not include the second format but includes the first format, and to store the backup result in the first format and the backup result in the second format into data tables of corresponding formats respectively, so as to obtain the backup history data corresponding to the backup result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images