Database migration management system and method based on cloud

Abstract

The invention discloses a cloud-based database migration management system and method, and belongs to the technical field of database migration management. The method comprises the following steps: extracting to-be-migrated data of a source database and preprocessing; obtaining timestamps of the to-be-migrated data recorded by the source database, and performing priority ranking on the to-be-migrated data; dividing a migration sequence of the to-be-migrated data and distributing the to-be-migrated data to a plurality of migration tasks; obtaining the current system load and the migration data volume of available storage and migration tasks, dynamically calculating the optimal migration cache size, and adjusting the calculation, storage and network distribution of migration resources; a parallel migration task is started, the state of the migration task is monitored in real time, errors in the migration process are recognized, and error information and recovery operation options are provided through a graphical user interface; the database migration efficiency and the resource utilization rate are improved, and the requirement for human intervention is lowered; and incremental migration and large-scale data migration are supported.

Description

Technical Field

[0001] This invention relates to the field of database migration management technology, specifically to a cloud-based database migration management system and method. Background Technology

[0002] With the rapid development of information technology, enterprises and organizations face a large number of data migration needs, especially in data center migration, cloud computing, and hybrid cloud environments. How to efficiently and accurately migrate data from traditional on-premises databases to cloud databases has become an urgent technical problem to be solved. Existing database migration methods often lack sufficient flexibility and customizability. Although common migration frameworks can perform large-scale data migration, they still suffer from inefficiency and high error rates when dealing with large data volumes, complex structures, and high-concurrency migration needs.

[0003] In existing technologies, database migration management is mostly coarse-grained, lacking precise control and dynamic adjustment of migration tasks. It typically relies on static settings to control migration tasks, failing to make flexible adjustments in real-time based on migration progress, resource consumption, and priority. Therefore, developing a fine-grained database migration management method, especially one that provides flexible and customizable interfaces based on specific task requirements, is urgently needed. Summary of the Invention

[0004] The purpose of this invention is to provide a cloud-based database migration management system and method to solve the problems mentioned in the background art.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A cloud-based database migration management system includes a migration task management module, a dynamic resource scheduling module, a graphical user interface module, a heterogeneous database interface module, and an error detection and recovery module. The migration task management module is used to schedule database migration tasks and sort them according to the priority of the migration data. The dynamic resource scheduling module is used to intelligently adjust the allocation of computing, storage, and network resources based on the current resource load of the cloud platform; A graphical user interface module is used to display the migration progress and allow users to monitor the task status in real time and intervene manually. The heterogeneous database interface module is used to support seamless migration between source and target databases and to perform data schema conversion. The error detection and recovery module is used to automatically detect migration errors during the migration process and provide error recovery operations.

[0006] It's important to explain that during the data migration process, the system can monitor the execution status of the migration task in real time through an intelligent error detection and recovery mechanism. Once an error is detected (such as network interruption or target database unavailability), the system can immediately display the error information through a graphical interface and automatically restart the migration task or guide the user through troubleshooting. This mechanism ensures the high availability and reliability of the data migration task.

[0007] As a preferred embodiment of the cloud-based database migration management system and method of the present invention, the migration task management module further includes: The priority sorting function is used to sort database records based on their timestamps, sizes, and business priorities. The migration queue management function is used to queue data to be migrated according to priority and assign it to different tasks for processing.

[0008] As a preferred embodiment of the cloud-based database migration management system and method of the present invention, the dynamic resource scheduling module monitors the resource load of the cloud platform, dynamically adjusts the resource allocation of each migration task, and adjusts the resource configuration in real time according to the migration progress.

[0009] As a preferred embodiment of the cloud-based database migration management system and method of the present invention, the graphical user interface module includes: The task list feature displays the current status, progress bar, and remaining migration data for all migration tasks. Error message function: When an error occurs during the migration task, the GUI will display error information and provide a recovery button; The manual intervention feature allows users to adjust the migration order based on task priority and manually pause or retry tasks.

[0010] As a preferred embodiment of the cloud-based database migration management system and method of the present invention, the heterogeneous database interface module is used to establish a mapping between data format and table structure.

[0011] As a preferred embodiment of the cloud-based database migration management system and method of the present invention, the error detection and recovery module is used to monitor the migration status of the migration task in real time. When an error occurs during the migration process, the module performs a task recovery operation and restarts the corresponding migration task.

[0012] A cloud database migration method, the method comprising the following steps: Extract the data to be migrated from the source database and preprocess it. The data to be migrated includes the difference data between the source database and the target database. Obtain the timestamps of the data to be migrated from the source database and prioritize the data to be migrated. Based on the priority ranking results, the migration order of the data to be migrated is determined and the data is assigned to multiple migration tasks; Obtain the current system load, available storage, and migration data volume of the migration task; dynamically calculate the optimal migration cache size; and adjust the computation, storage, and network allocation of migration resources. Start the parallel migration task and monitor the migration task status in real time; It's important to explain that through dynamic resource scheduling, the system can intelligently calculate and adjust resource allocation during the migration process. It not only supports migration of single tasks but also efficiently handles multiple migration tasks through parallel migration technology. This effectively solves the performance bottleneck of traditional single-threaded migration tools when facing large-scale data migration, enabling the system to fully utilize the computing and storage resources of the cloud platform and improve resource utilization.

[0013] Based on the migration task status, errors during the migration process are identified and error information and recovery options are provided through a graphical user interface.

[0014] As a preferred embodiment of the cloud-based database migration management system and method of the present invention, the specific implementation process of extracting the data to be migrated from the source database and preprocessing it, wherein the data to be migrated includes the difference data between the source database and the target database, includes: The incremental migration function is used to obtain the difference data between the source database and the target database, which is then used as the data to be migrated. Obtain the data schemas of the source and target databases, establish a mapping between the data formats and table structures in the source and target data schemas, and remove data from the data to be migrated that cannot be mapped.

[0015] As a preferred embodiment of the cloud-based database migration management system and method of the present invention, the specific implementation process of obtaining the timestamp of the data to be migrated recorded in the source database and prioritizing the data to be migrated includes: Obtain the timestamps of the data to be migrated from the source database, filter the data to be migrated whose timestamps are greater than the time threshold, sort the filtered data to be migrated in descending order of timestamp size, and construct a migration sequence. When the timestamps of the data to be migrated in the migration dataset are the same size, sort the data with the same timestamps in ascending order according to data size, and update the migration sequence. Where N represents the total number of data to be migrated. This represents the nth piece of data to be migrated; Based on the index of the data to be migrated in the migration sequence, the priority of the data to be migrated is quantified, and the replanting of the index value as a percentage of the total index values ​​is recorded as the priority of the data to be migrated. Extract the business requirements and urgency of the data to be migrated, weight the data to be migrated in the migration sequence to obtain the priority of the weighted data to be migrated; sort the data to be migrated in descending order of priority and update the migration sequence.

[0016] As a preferred embodiment of the cloud-based database migration management system and method of the present invention, the specific implementation process of obtaining the current system load, available storage and migration data volume of migration tasks, dynamically calculating the optimal migration cache size, and adjusting the calculation, storage and network allocation of migration resources includes: Get the current system load, available storage, and migration data volume of the migration task. When the current system load is less than the load threshold, increase the cache by 10%; when the current system load is greater than or equal to the load threshold, use the baseline cache.

[0017] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: The cloud-based database migration management system and method provided by this invention extracts the data to be migrated from the source database and performs preprocessing; obtains the timestamps of the data to be migrated recorded in the source database and prioritizes the data; divides the migration order of the data to be migrated and allocates it to multiple migration tasks; obtains the current system load, available storage, and migration data volume of the migration tasks, dynamically calculates the optimal migration cache size, and adjusts the computation, storage, and network allocation of migration resources; initiates parallel migration tasks, monitors the migration task status in real time, identifies errors during the migration process, and provides error information and recovery operation options through a graphical user interface; improves the efficiency of database migration and resource utilization, reduces the need for human intervention; and supports incremental migration and large-scale data migration. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0019] Figure 1 This is a schematic diagram of the system architecture in an embodiment of the present invention; Figure 2 This is a flowchart of the database migration process in an embodiment of the present invention; Figure 3 This is a schematic diagram of dynamic resource scheduling and parallel migration in an embodiment of the present invention; Figure 4 This is a schematic diagram of schema conversion between heterogeneous databases in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 In this first embodiment: a cloud-based database migration management system, the system includes a migration task management module, a dynamic resource scheduling module, a graphical user interface module, a heterogeneous database interface module, and an error detection and recovery module. The migration task management module is used to schedule database migration tasks and sort them according to the priority of the migration data. Unlike existing technologies, this invention automatically optimizes the execution order and priority of migration tasks through an intelligent scheduling algorithm and adopts a dynamic task scheduling mechanism based on database record timestamps and data volume to improve migration efficiency.

[0022] The dynamic resource scheduling module is used to intelligently adjust the allocation of computing, storage, and network resources based on the current resource load of the cloud platform; A graphical user interface (GUI) module is used to display the migration progress and allow users to monitor task status and intervene manually in real time. This invention provides a dynamically updated GUI interface that allows users to view the migration progress in real time and flexibly adjust task priorities and perform error recovery operations. This interface enables full interaction between the user and the database migration process, filling the gap in traditional tools where users cannot intervene and adjust migration tasks in real time. Unlike traditional migration tools that can only passively monitor migration progress, the system of this invention achieves dynamic task adjustment and progress feedback through a graphical user interface (GUI). Users can view the progress of each migration task in real time and manually adjust task priorities or restart failed tasks through the interface. During the migration process, if errors such as network interruption or target database unavailability are encountered, the system can immediately prompt through the interface and automatically resume the migration task. The specific code is as follows: def monitor_progress(task_id): try: # Monitor the task progress via API task_progress = check_task_progress(task_id) if task_progress['status'] == 'error': handle_error(task_id) else: return task_progress['percentage'] except NetworkError: The function returns 'Waiting for recovery'. This feature allows users to proactively intervene in the migration process, ensuring the entire migration task is completed as planned.

[0023] This invention provides a heterogeneous database interface module to support seamless migration between source and target databases and to perform data schema conversion. It supports seamless migration between heterogeneous database systems, including but not limited to relational databases (such as MySQL and PostgreSQL) and non-relational databases (such as MongoDB). Compared to existing products, this invention's interface module can intelligently identify structural differences between the source and target databases and automatically adjust data migration strategies, reducing the probability of manual intervention and errors.

[0024] The error detection and recovery module is used to automatically detect migration errors during the migration process and provide error recovery operations.

[0025] Specifically, the migration task management module also includes: The priority sorting function is used to sort database records based on their timestamps, sizes, and business priorities. The migration queue management function is used to queue data to be migrated according to priority and assign it to different tasks for processing.

[0026] Specifically, the dynamic resource scheduling module monitors the resource load of the cloud platform, dynamically adjusts the resource allocation for each migration task, and adjusts the resource configuration in real time according to the migration progress.

[0027] Specifically, the graphical user interface module includes: The task list feature displays the current status, progress bar, and remaining migration data for all migration tasks. Error message function: When an error occurs during the migration task, the GUI will display error information and provide a recovery button; The manual intervention feature allows users to adjust the migration order based on task priority and manually pause or retry tasks.

[0028] Specifically, the heterogeneous database interface module is used to establish a mapping between data formats and table structures.

[0029] Specifically, the error detection and recovery module is used to monitor the migration status of the migration task in real time. When an error occurs during the migration process, it performs a task recovery operation and restarts the corresponding migration task.

[0030] Please see Figure 2 In this second embodiment, a cloud database migration method is provided, the method comprising the following steps: Extract the data to be migrated from the source database and preprocess it. The data to be migrated includes the difference data between the source database and the target database. Obtain the timestamps of the data to be migrated from the source database and prioritize the data to be migrated. Based on the priority ranking results, the migration order of the data to be migrated is determined and the data is assigned to multiple migration tasks; Obtain the current system load, available storage, and migration data volume of the migration task; dynamically calculate the optimal migration cache size; and adjust the computation, storage, and network allocation of migration resources. In existing technologies, database migration tools are mostly single-threaded or have fixed resource allocation, failing to automatically allocate computing and storage resources according to the different needs of migration tasks. This invention, through a cloud resource management module, can dynamically allocate computing resources based on the size and priority of migration tasks, and perform parallel processing of tasks on the cloud platform. The code is as follows: def distribute_resources(tasks, available_resources): task_resources = {} For task in tasks: task_resources[task['id']] = available_resources * task['priority'] / sum([task['priority'] for task in tasks]) The system intelligently schedules tasks based on task priority and resource availability, ensuring optimal resource utilization during large-scale data migration and avoiding performance bottlenecks caused by single-threaded migration.

[0031] Start the parallel migration task and monitor the migration task status in real time; Based on the migration task status, errors during the migration process are identified and error information and recovery options are provided through a graphical user interface.

[0032] Specifically, the process of extracting and preprocessing the data to be migrated from the source database, where the data to be migrated includes the difference data between the source database and the target database, includes: The incremental migration function is used to obtain the difference data between the source database and the target database, which is then used as the data to be migrated. Obtain the data schemas of the source and target databases, establish a mapping between the data formats and table structures in the source and target data schemas, and remove data from the data to be migrated that cannot be mapped.

[0033] This invention supports automatic data schema conversion even when the source and target databases have inconsistent structures. Traditional migration tools are typically only suitable for migrating databases of the same type, while this invention can intelligently adjust the data migration strategy according to the characteristics of different databases, automatically handling data type conversion and table structure mapping, greatly reducing data incompatibility issues during migration. The code is as follows: def schema_conversion(source_record, target_schema): # Convert source record to match target schema converted_record = {} for field in source_record: if field in target_schema: converted_record[target_schema[field]] = source_record[field] The `converted_record` function returns a `transformed_record`. This data conversion mechanism ensures smooth data migration between the source and target databases in heterogeneous database environments without manual intervention.

[0034] like Figure 4 This demonstrates how seamless migration can be achieved using the schema conversion module of this invention when the source and target databases have inconsistent schemas. The figure shows the automatic conversion process of data from the source database format to the target database.

[0035] For example, the name and data type of a field in the source database may differ from those in the target database. The system uses schema conversion to map the data in the source database to the correct format and data type in the target database, ensuring data consistency during the migration process.

[0036] Specifically, the specific implementation process of obtaining the timestamps of the data to be migrated from the source database and prioritizing the data to be migrated includes: Obtain the timestamps of the data to be migrated from the source database, filter the data to be migrated whose timestamps are greater than the time threshold, sort the filtered data to be migrated in descending order of timestamp size, and construct a migration sequence. When the timestamps of the data to be migrated in the migration dataset are the same size, sort the data with the same timestamps in ascending order according to data size, and update the migration sequence. Where N represents the total number of data to be migrated. This represents the nth piece of data to be migrated; Based on the index of the data to be migrated in the migration sequence, the priority of the data to be migrated is quantified, and the replanting of the index value as a percentage of the total index values ​​is recorded as the priority of the data to be migrated. Extract the business requirements and urgency of the data to be migrated, weight the data to be migrated in the migration sequence to obtain the priority of the weighted data to be migrated; sort the data to be migrated in descending order of priority and update the migration sequence.

[0037] Traditional migration task scheduling typically relies on fixed strategies, which cannot effectively adapt to dynamically changing migration task requirements. This invention introduces an intelligent sorting mechanism based on the timestamps and sizes of database records during task scheduling, combined with a dynamic adjustment strategy for data block priorities, achieving efficient and on-demand resource allocation. The specific implementation code is as follows: def prioritize_records(records, migration_time_threshold): # Newer records with smaller size are given higher priority records_sorted_by_time = sorted(records, key=lambda x: (x['timestamp'], x['size']), reverse=True) The `return [record for record in records_sorted_by_time if record['timestamp'] > migration_time_threshold]` method, based on real-time data stream sorting, ensures the efficiency of migration tasks, especially when dealing with large amounts of data, automatically prioritizing the migration of more urgent data.

[0038] like Figure 3 This diagram illustrates how a database migration management tool enables parallel processing of migration tasks through a cloud resource scheduling module. The diagram shows how multiple migration tasks are executed in parallel on the cloud platform, and how the system automatically adjusts the allocation of computing resources based on the priority, data volume, and resource requirements of each task.

[0039] The diagram also shows the dynamic adjustment of task allocation. For example, when some tasks have higher priority, they will be allocated more computing resources, while tasks with fewer resources will be migrated as needed.

[0040] Specifically, the implementation process of obtaining the current system load, available storage, and migration data volume of the migration task, dynamically calculating the optimal migration cache size, and adjusting the calculation, storage, and network allocation of migration resources includes: Get the current system load, available storage, and migration data volume of the migration task. When the current system load is less than the load threshold, increase the cache by 10%; when the current system load is greater than or equal to the load threshold, use the baseline cache.

[0041] In existing technologies, the cache size for database migration tasks is often statically set and cannot be adjusted according to real-time load. This invention employs a dynamic cache adjustment algorithm based on cloud resource conditions. It intelligently calculates the optimal cache size based on the current system load, available storage, and the amount of data to be migrated, thereby avoiding the problems of insufficient cache or resource waste during the migration process. The code is as follows: def calculate_optimal_cache_size(total_available_space, current_load,migration_percentage=0.7): if current_load < 50: # If system load is low, increase cache size return total_available_space * (migration_percentage + 0.1) else: # Adjust cache size based on system load This method returns `total_available_space * migration_percentage`. It automatically adjusts the cache size based on real-time load monitoring, ensuring efficient execution of migration tasks without impacting system performance.

[0042] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0043] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cloud-based database migration management system, characterized in that, The system includes a migration task management module, a dynamic resource scheduling module, a graphical user interface module, a heterogeneous database interface module, and an error detection and recovery module. The migration task management module is used to schedule database migration tasks and sort them according to the priority of the migration data. The dynamic resource scheduling module is used to intelligently adjust the allocation of computing, storage, and network resources based on the current resource load of the cloud platform; A graphical user interface module is used to display the migration progress and allow users to monitor the task status in real time and intervene manually. The heterogeneous database interface module is used to support seamless migration between source and target databases and to perform data schema conversion. The error detection and recovery module is used to automatically detect migration errors during the migration process and provide error recovery operations.

2. The cloud-based database migration management system according to claim 1, characterized in that, The migration task management module also includes: The priority sorting function is used to sort database records based on their timestamps, sizes, and business priorities. The migration queue management function is used to queue data to be migrated according to priority and assign it to different tasks for processing.

3. The cloud-based database migration management system according to claim 1, characterized in that, The dynamic resource scheduling module monitors the resource load of the cloud platform, dynamically adjusts the resource allocation for each migration task, and adjusts the resource configuration in real time according to the migration progress.

4. The cloud-based database migration management system according to claim 1, characterized in that, The graphical user interface module includes: The task list feature displays the current status, progress bar, and remaining migration data for all migration tasks. Error message function: When an error occurs during the migration task, the GUI will display error information and provide a recovery button; The manual intervention feature allows users to adjust the migration order based on task priority and manually pause or retry tasks.

5. A cloud-based database migration management system according to claim 1, characterized in that, The heterogeneous database interface module is used to establish a mapping between data format and table structure.

6. The cloud-based database migration management system according to claim 1, characterized in that, The error detection and recovery module is used to monitor the migration status of the migration task in real time. When an error occurs during the migration process, it performs a task recovery operation and restarts the corresponding migration task.

7. A cloud-based database migration method, using a cloud-based database migration management system as described in any one of claims 1-6, characterized in that, The method includes the following steps: Extract the data to be migrated from the source database and preprocess it. The data to be migrated includes the difference data between the source database and the target database. Obtain the timestamps of the data to be migrated from the source database and prioritize the data to be migrated. Based on the priority ranking results, the migration order of the data to be migrated is determined and the data is assigned to multiple migration tasks; Obtain the current system load, available storage, and migration data volume of the migration task; dynamically calculate the optimal migration cache size; and adjust the computation, storage, and network allocation of migration resources. Start the parallel migration task and monitor the migration task status in real time; Based on the migration task status, errors during the migration process are identified and error information and recovery options are provided through a graphical user interface.

8. A cloud database migration method according to claim 7, characterized in that: The specific implementation process of extracting and preprocessing the data to be migrated from the source database, wherein the data to be migrated includes the difference data between the source database and the target database, includes: The incremental migration function is used to obtain the difference data between the source database and the target database, which is then used as the data to be migrated. Obtain the data schemas of the source and target databases, establish a mapping between the data formats and table structures in the source and target data schemas, and remove data from the data to be migrated that cannot be mapped.

9. A cloud database migration method according to claim 8, characterized in that: The specific implementation process of obtaining the timestamps of the data to be migrated from the source database and prioritizing the data to be migrated includes: Obtain the timestamps of the data to be migrated from the source database, filter the data to be migrated whose timestamps are greater than the time threshold, sort the filtered data to be migrated in descending order of timestamp size, and construct a migration sequence. When the timestamps of the data to be migrated in the migration dataset are the same size, sort the data with the same timestamps in ascending order according to data size, and update the migration sequence. Where N represents the total number of data to be migrated. This represents the nth piece of data to be migrated; Based on the index of the data to be migrated in the migration sequence, the priority of the data to be migrated is quantified, and the replanting of the index value as a percentage of the total index values ​​is recorded as the priority of the data to be migrated. Extract the business requirements and urgency of the data to be migrated, weight the data to be migrated in the migration sequence to obtain the priority of the weighted data to be migrated; sort the data to be migrated in descending order of priority and update the migration sequence.

10. A cloud database migration method according to claim 9, characterized in that: The specific implementation process of obtaining the current system load, available storage, and migration data volume of the migration task, dynamically calculating the optimal migration cache size, and adjusting the computation, storage, and network allocation of migration resources includes: Get the current system load, available storage, and migration data volume of the migration task. When the current system load is less than the load threshold, increase the cache by 10%. When the current system load is greater than or equal to the load threshold, the baseline cache is used.

Images