Non-perception data dynamic migration method

Abstract

The invention provides a non-perceptual data dynamic migration method which comprises the following steps: acquiring a difference between a source database and a target database, and making a data lossless migration plan; comparing the data difference between the source database and the target database, judging the data volume needing to be synchronized each time, and predicting the required network bandwidth resources; estimating transmission time sequences and transmission paths of different data blocks by using a Kalman filtering algorithm according to importance and priority of an application system and data; continuously monitoring data synchronization quality and application system performance indexes in a data migration task execution process, and judging a smooth transition effect; after core service data migration is finished, paused non-core services are recovered step by step until migration and synchronization of all service data of the standby data center are finished; a gray release mode of gradually expanding the weight is adopted, the service is smoothly transited to a data center, and a one-key automatic full-service environment rollback scheme is established to prevent abnormity.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a method for dynamic migration of imperceptible data. Background Art

[0002] With the continuous growth of data volume and the continuous expansion of business, the real-time migration of massive data has become a critical issue. However, it is not easy to improve the real-time zero-loss migration of massive data while ensuring that the performance of the application system does not degrade. Traditional data migration methods often require a large amount of system resources and network bandwidth, resulting in a decrease in the performance of the application system. At the same time, due to limited network bandwidth, the real-time migration of high-priority core business data is also difficult. During the migration process, system stability and data consistency are extremely important. Therefore, it is necessary to achieve lossless data migration and ensure zero-error transmission of core business data. At the same time, in order to smoothly transition to the new system, after the migration is complete, users need to use the new system unconsciously without any interruption or impact to the business. In addition, a smooth and imperceptible transition to the new system after migration, and the ability to quickly and automatically fall back to the original system in the event of failure, is also a challenging problem. Summary of the Invention

[0003] The present invention provides a method for dynamic data migration without perception, which mainly includes:

[0004] Obtain the differences between the source database and the target database and formulate a lossless data migration plan; compare the data differences between the source database and the target database, determine the amount of data that needs to be synchronized each time, and predict the required network bandwidth resources; if the current network bandwidth resources can meet the data migration needs, apply to expand the corresponding storage capacity and computing processing power as a backup resource pool for subsequent data migration; based on the importance and priority of the application system and data, use the Kalman filter algorithm to estimate the transmission time series and transmission path of different data blocks; continuously monitor the data synchronization quality and application system performance indicators during the execution of the data migration task to determine the effect of the smooth transition; if a network bottleneck occurs during data migration, suspend non-core businesses according to the priority of the business and release the resources occupied by non-core businesses to ensure that the core business data is migrated; wait for the core business data migration to be completed, and gradually resume the suspended non-core businesses until all business data migration and synchronization of the backup data center are completed; adopt a grayscale release model with gradually increasing weights to smoothly transition the business to the data center, and establish a one-click automated full business environment fallback plan to prevent abnormalities.

[0005] Furthermore, obtaining the differences between the source database and the target database and formulating a lossless data migration plan includes:

[0006] Obtain the differences between the table structures of the source and target databases, including differences in table names, column names, data types, primary keys, and indexes; if there are differences in the table structures between the source and target databases, establish a field mapping relationship and convert the target database table structure; obtain incremental data changes between the source and target databases, including: using a polling method to capture the add, delete, and modify operations of the source database in real time; based on the table structures and field mapping relationships between the source and target databases, convert the incremental data into a format and write it to the target database; formulate a data lossless migration plan, including: continuously tracking the data differences between the source and target databases during the data migration process If there is a difference between the source and target databases, determine whether there is any remaining data that needs to be migrated. If the business data of the source and target databases are related, sort them according to the relationship and prioritize migrating the related or dependent business data. If there are differences in the index structure between the source and target databases, you need to re-index the target database tables based on the target database's index structure after the migration is complete. If there are differences in the constraints between the source and target databases, you need to gradually add constraints to the target database tables after the migration is complete. If there are differences in the custom functions between the source and target databases, you need to gradually add custom functions to the target database after the migration is complete.

[0007] Furthermore, the step of comparing the data differences between the source database and the target database, determining the amount of data to be synchronized each time, and predicting the required network bandwidth resources includes:

[0008] Determine whether there are differences in the table structures between the source and target databases, and calculate the data volume of all tables to be migrated in the source database, including: if the table structures between the source and target databases are exactly the same, obtain the current data volume of the target database table; if there are differences in the table structures between the source and target databases, perform conversion calculations based on the field mapping relationship; calculate the amount of data that has been migrated historically; calculate the remaining amount of data to be migrated by subtracting the migrated amount from the total data volume; obtain the historical data change volume and predict the peak situation of synchronized data through a linear regression algorithm; calculate the required network bandwidth resources based on the network bandwidth calculation formula, combined with the peak data volume and the acceptable transmission time threshold.

[0009] Furthermore, if the current network bandwidth resources can meet the data migration requirements, then applying to expand the corresponding storage capacity and computing processing capabilities as a backup resource pool for subsequent data migration includes:

[0010] Determine the degree of redundancy of spare capacity and resources based on the availability requirements of the business; use a linear regression prediction algorithm to predict spare storage capacity requirements based on the business data volume and data growth rate; establish a spare capacity assessment model based on the degree of redundancy of spare capacity and resources and the predicted spare storage capacity requirements; based on the established spare capacity assessment model, allocate the size of spare storage capacity and computing resources according to the priority and importance of the business system; if it is assessed that the current network bandwidth resources can meet the expected data migration needs, then implement a data redundancy backup plan to expand the target based on the storage space requirements and computing processing power requirements after the subsequent migration is completed The capacity of the data center includes: expanding storage space and improving computing power based on the spare capacity assessment model; increasing resource monitoring and alarms; expanding storage space by adding physical machines or using virtualization components as a spare resource pool; computing power by vertically expanding existing physical machines or horizontally adding new servers; among them, increasing resource monitoring and alarms includes: determining whether there are faulty resources in the spare resource pool and recording the fault status; obtaining resource utilization history records through the management system of the spare resource pool; determining whether the resource utilization rate exceeds the preset threshold, and if so, triggering the corresponding alarm; determining the alarm notification method, and sending the alarm notification to relevant personnel.

[0011] Furthermore, the Kalman filter algorithm is used to estimate the transmission time sequence and transmission path of different data blocks according to the importance and priority of the application system and data, including:

[0012] Determine the priority based on the importance of the business system. The system with the highest priority has the highest weight for data migration. Divide the business database into multiple data blocks, and mark each data block in turn according to importance, priority, and business relevance. Analyze the candidate transmission paths of different data blocks, and use the Kalman filter algorithm to estimate the candidate transmission paths of different data blocks. Determine the logical relationship between different data blocks, use the Kalman filter algorithm to integrate the data dependency between different data blocks, and calculate an optimal transmission time series based on the principle that high-priority data blocks are transmitted before low-priority data blocks and the logical relationship between different data blocks.

[0013] Furthermore, the continuous monitoring of data synchronization quality and application system performance indicators during the execution of the data migration task to determine the smooth transition effect includes:

[0014] After starting lossless data migration, compare the source data and target data in real time to determine whether there are inconsistencies or losses between the source and target data; at the same time, verify the integrity of the migrated data to prevent omissions of some fields or records; obtain the data transmission delay, data processing delay and data update delay of the source and target systems, as well as the data integrity error rate, data consistency error rate and data format error rate during data transmission; monitor the data synchronization quality in real time by recording the number of records or bytes transmitted; compare the differences in response time and throughput performance indicators of business applications in the source center system and the target system to determine the effect of the smooth transition, including: comparing the response time data of the source and target system business applications under the same load to determine the difference in response time; comparing the throughput data of the source and target system business applications under the same load to determine the difference in throughput; comparing the error rate data of the source and target system business applications under the same load to determine the difference in error rate; comparing the availability of the source and target system business applications under the same load to determine the difference in availability.

[0015] Furthermore, if a network bottleneck occurs during data migration, non-core services are suspended according to service priority to release resources occupied by non-core services to ensure that the data of core services is migrated, including:

[0016] If during the migration process, it is detected that the core data migration link is congested and delayed, and the predetermined network performance index requirements cannot be met, then it is determined that a network bottleneck has occurred in the data migration; wherein, the services are divided into high-priority core services and low-priority non-core services based on priority; when a network bottleneck occurs in the data migration, the non-core services are suspended according to the service priority and the computing resources occupied by them are released, including: sorting the services according to the importance and urgency of the services and determining the service priority; obtaining the dependency relationship of each service and judging whether suspending the low-priority non-core services will affect the dependent services; if not, marking the service as suspendable; checking the resource occupancy details of each service and starting the backup resource pool; if starting the backup resource pool still cannot meet the predetermined network performance index requirements, suspending the services marked as suspendable and releasing the computing resources occupied by them.

[0017] Furthermore, after waiting for the core business data migration to be completed, the suspended non-core business is gradually resumed until all business data of the standby data center are migrated and synchronized, including:

[0018] After waiting for the high-priority core business data migration to be completely completed, verify the progress monitoring and data integrity to confirm that the core business has been fully synchronized to the target data center; obtain business associations, identify and gradually restore non-core businesses that have no dependencies on the core business; among them, restoring non-core businesses that have no dependencies on the core business includes: verifying before restoring non-core businesses to ensure that no service data is missing or status errors occur during the suspension of non-core businesses; if there is any loss of incremental data, retrieve the lost data from the source database; after restoring non-core businesses, verify business test cases to ensure that all businesses are completely switched and successfully deployed in the target data center, and that the source and target databases are consistent.

[0019] Furthermore, the adoption of a phased release model with gradually increased weighting allows for a smooth transition of services to the data center, and establishes a one-click automated fallback solution for the entire business environment to prevent anomalies, including:

[0020] Gradually release the business system in grayscale according to the priority from high to low, gradually expand the weight of the new data center, and smoothly transition the business to the data center; do not interrupt the user's ongoing business operation sessions while graying out the business; monitor real-time business traffic at the same time, and issue early warning notifications if any anomalies are found; if an early warning notification is received, determine which link has the anomaly and make business adjustments based on the judgment results; establish a one-click automated full-business environment rollback plan, which can quickly roll back all businesses in the new data center to the original production environment at one time.

[0021] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0022] The present invention discloses a method for imperceptible dynamic data migration. By comparing the data differences between the source database and the target database, the amount of data that needs to be synchronized each time can be determined, and the required network bandwidth resources can be predicted. When the current network bandwidth resources can meet the data migration needs, an application is made to expand the corresponding storage capacity and computing processing capabilities as a backup resource pool for subsequent data migration. According to the importance and priority of the application system and data, the Kalman filter algorithm can be used to estimate the transmission time series and transmission paths of different data blocks. If a network bottleneck occurs in data migration due to special reasons, non-core businesses can be suspended according to the priority of the business, and the resources occupied by the non-core business can be released to ensure that the data of the core business can be migrated with zero errors. High-speed and real-time lossless data migration is achieved, the stability of zero-error transmission of core business data is ensured, and a smooth and imperceptible transition to the new system can be made while providing rapid and automatic rollback in the event of a failure, thereby ensuring the smooth progress of system migration and upgrade. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flowchart of a method for dynamic data migration without perception of the present invention.

[0024] Figure 2 A schematic diagram of a method for dynamic data migration without perception according to the present invention. DETAILED DESCRIPTION

[0025] The following will describe the technical solutions in the embodiments of the present invention in detail with reference to the accompanying drawings. The described embodiments are only a part of the embodiments of the present invention.

[0026] In this embodiment, a method for dynamic data migration without perception may specifically include:

[0027] S101. Obtain the differences between the source database and the target database and develop a data lossless migration plan.

[0028] Obtain differences between the source and target databases' table structures, including differences in table names, column names, data types, primary keys, and indexes. If there are differences in the source and target databases' table structures, establish field mappings and convert the target database's table structure. Obtain incremental data changes between the source and target databases, including using polling to capture add, delete, and modify operations in real time. Based on the source and target databases' table structures and field mappings, convert the incremental data before writing it to the target database. Develop a lossless data migration plan, including continuously tracking data differences between the source and target databases during the migration process to determine whether there is any remaining data to migrate. If the business data in the source and target databases is related, prioritize the migration based on the related relationships, prioritizing related or dependent business data. If there are differences in the index structure between the source and target databases, re-index the target database's tables based on the target database's index structure after the migration is complete. If there are differences in constraints between the source and target databases, gradually add constraints to the target database's tables after the migration is complete. If there are differences in the custom functions between the source and target databases, you need to gradually add custom functions to the target database after the migration is complete.

[0029] For example, the source database has a table named users with the following columns: id, name, and age, which are integer, string, and integer, respectively. The target database also has a table named users, but with a slightly different column structure: id, full_name, age, and gender, which are integer, string, integer, and string, respectively. You can create field mappings to map the source database's name column to the target database's full_name column, and the source database's age column to the target database's age column. You can convert the data format in the source database and write it to the target database. Furthermore, using a polling method, you can capture add, delete, and modify operations in the source database in real time. For example, a new record is added to the source database with an id of 1, a name of John, and an age of 25. During the data migration process, you can pre-plan a lossless data migration. For example, periodically compare the data differences between the source and target databases to determine whether there is any remaining data to migrate. If so, proceed with the data migration. If the index structures of the source and target databases differ, you can re-index the target database's tables after the migration is complete, based on the target database's index structure. Similarly, if there are differences in constraints, you can gradually add the target database's constraints after the migration is complete to avoid impacting online business. If there are differences in custom functions between the source and target databases, you can also gradually add the target database's custom functions after the migration is complete.

[0030] S102: Compare the data differences between the source database and the target database, determine the amount of data that needs to be synchronized each time, and predict the required network bandwidth resources.

[0031] Determine whether there are any differences in the table structures between the source and target databases, and calculate the data volume of all tables to be migrated in the source database. This includes: if the table structures between the source and target databases are exactly the same, obtain the current data volume of the target database table; if the table structures between the source and target databases differ, perform conversion calculations based on field mapping relationships; calculate the amount of data that has been migrated historically; and calculate the remaining amount of data to be migrated by subtracting the amount of data that has been migrated from the total amount of data. Obtain historical data changes and use a linear regression algorithm to predict the peak value of synchronized data. Calculate the required network bandwidth resources based on the network bandwidth calculation formula, combining the peak data volume and the acceptable transmission time threshold.

[0032] For example, the source database has three tables to be migrated: Table A, Table B, and Table C, with data volumes of 1,000, 2,000, and 3,000 records, respectively. The source and target databases have identical table structures. The current data volumes of Table A, Table B, and Table C in the target database are 500, 1,000, and 1,500, respectively. The previously migrated data volumes are 500, 800, and 1,000. The remaining data volume for Table A is 1,000 - 500 = 500. The remaining data volume for Table B is 2,000 - 1,000 = 1,000. The remaining data volume for Table C is 3,000 - 1,500 = 1,500. The differences in the table structures between the source and target databases necessitate conversion calculations for field mapping relationships. Table A's field mapping is one-to-one, Table B's field mapping is one-to-many, meaning one source field corresponds to multiple target fields. Table C's field mapping is many-to-one, meaning multiple source fields correspond to one target field. Table A has two fields, Table B has three source fields and two target fields, and Table C has four source fields and one target field. Each field is an integer. Based on the field mappings, data conversion calculations are performed. Table A's data volume remains unchanged at 500 records. Table B has 1000 records, but each source field is converted to two target fields using a one-to-many relationship, resulting in a target field volume of 1000 * 2 = 2000 records. Table C has 1500 records, but each source field is converted to one target field using a many-to-one relationship, resulting in a target field volume of 1500 / 4 = 375 records. The most recent real-time data change was 100 records. Based on historical data change trends, a linear regression algorithm predicts the next data change will be 120 records. If the acceptable transmission time threshold is 1 hour, or 60 minutes, then bandwidth = data volume / transmission time * 8. Bandwidth is measured in Mbps, data volume in MB, and transmission time in minutes. The peak data volume is 120 records, each 1 KB in size. The transmission time is 60 minutes. Bandwidth = 120 * 0.01 / 60 * 8 = 16 Mbps. Therefore, the required network bandwidth is 16 Mbps.

[0033] S103. If the current network bandwidth resources can meet the data migration requirements, apply to expand the corresponding storage capacity and computing processing capabilities as a backup resource pool for subsequent data migration.

[0034] Determine the degree of redundancy of spare capacity and resources based on the availability requirements of the business. Use a linear regression prediction algorithm to predict spare storage capacity requirements based on the data volume and data growth rate of the business. Establish a spare capacity assessment model based on the degree of redundancy of spare capacity and resources and the predicted spare storage capacity requirements. Based on the established spare capacity assessment model, allocate the size of spare storage capacity and computing resources according to the priority and importance of the business system. If it is assessed that the current network bandwidth resources can meet the expected data migration needs, then implement a data redundancy backup plan to expand the capacity of the target data center based on the storage space requirements and computing processing power requirements after the subsequent migration is completed, including: expanding storage space and improving computing power based on the spare capacity assessment model; and increasing resource monitoring and alarms. Expanding storage space means adding physical machines or using virtualization components as a spare resource pool. Computing power means vertically expanding existing physical machines or horizontally adding new servers. Among them, adding resource monitoring and alarms includes: determining whether there are faulty resources in the backup resource pool and recording the fault status; obtaining resource utilization history records through the backup resource pool management system; determining whether the resource utilization rate exceeds the preset threshold, and if so, triggering the corresponding alarm; determining the alarm notification method, and sending the alarm notification to relevant personnel.

[0035] For example, a company has three businesses: A, B, and C. Based on the importance and impact of each business, the priorities of A, B, and C are determined as follows: Business A is highly important and has a high impact, with a priority of 10; Business B is medium important and has a moderate impact, with a priority of 5; and Business C is low important and has a low impact, with a priority of 2. Based on the priorities of A, B, and C, the weights for allocating spare capacity are determined as follows: Business A has a priority of 10 and a weight of 4; Business B has a priority of 5 and a weight of 3; and Business C has a priority of 2 and a weight of 3. Next, the spare capacity and resource redundancy levels are determined based on the availability requirements of the businesses. Business A requires high availability and therefore a high level of redundancy, while Businesses B and C require medium and low availability and therefore require a low level of redundancy. The redundancy of spare capacity and resources for Service A is high (high redundancy means 150% of the original capacity); Service B is medium (medium redundancy means 120% of the original capacity); and Service C is low (low redundancy means 100% of the original capacity). Then, a linear regression prediction algorithm is used to predict spare storage capacity requirements based on the data volume and data growth rate of each service. Based on historical data, the following prediction results are obtained: Service A has a data volume of 1000GB and an annual growth rate of 20%, resulting in a predicted spare storage capacity requirement of 1200GB; Service B has a data volume of 500GB and an annual growth rate of 10%, resulting in a predicted spare storage capacity requirement of 550GB; and Service C has a data volume of 200GB and an annual growth rate of 5%, resulting in a predicted spare storage capacity requirement of 210GB. A spare capacity assessment model is established based on the spare capacity allocation weights, the spare capacity and resource redundancy, and the predicted spare storage capacity requirements. The assessment indicates that the current network bandwidth resources can meet the projected data migration needs of System A, so the capacity of the target data center can be expanded. Based on the storage space and computing power requirements after the migration is complete, it was decided to expand storage space and increase computing power. For storage expansion, the decision was made to add virtualized component storage. System A currently has 100 TB of storage space. Based on the capacity assessment model, an additional 50 TB of storage space is needed to meet future needs. To increase computing power, new servers were added horizontally. System A currently has 10 servers, and based on the capacity assessment model, an additional 5 servers are needed to increase computing power. Resource monitoring and alarm units can also be added. If a server in the backup resource pool fails, the system will record the server's failure status. The backup resource pool management system can analyze resource utilization over the past week. If resource utilization exceeds 90%, the system will trigger an alarm. Alarm notifications can be sent to system administrators via email or text message.

[0036] S104. Based on the importance and priority of the application system and data, use the Kalman filter algorithm to estimate the transmission time sequence and transmission path of different data blocks.

[0037] Prioritize business systems based on their importance. The highest-priority system will receive the highest weight for data migration. Divide the business database into multiple data blocks, labeling each block according to importance, priority, and business relevance. Analyze candidate transmission paths for different data blocks and use the Kalman filter algorithm to estimate them. Determine the logical relationships between different data blocks. Apply the Kalman filter algorithm to integrate the data dependencies between different data blocks. Calculate an optimal transmission time sequence based on the principle that high-priority data blocks are transmitted before low-priority data blocks and the logical relationships between different data blocks.

[0038] For example, consider an e-commerce business system that includes three key modules: order management, inventory management, and user management. Data migration priorities must be determined to ensure the proper operation of the system. The business database can be divided into three data blocks: order data, inventory data, and user data. These data blocks can be labeled based on importance, priority, and business relevance. Order data is the most important, directly related to the core business, and therefore has the highest priority. Inventory data is the next most important, directly impacting order processing. User data is relatively less important, as it is primarily used for user management and statistical analysis. Next, candidate transmission paths for these data blocks are analyzed. Order data and inventory data have a dependency, while user data has no direct dependency on either order or inventory data. This means that order and inventory data can be transmitted simultaneously, while user data can be transmitted after the order and inventory data are transmitted. To estimate candidate transmission paths for different data blocks, a Kalman filter algorithm can be used. This algorithm estimates the position and velocity of an object by fusing data from different sensors. The transmission paths for different data blocks can be treated as separate sensor data, and the Kalman filter algorithm can be used to estimate their transmission times. Finally, the logical relationships between the different data blocks are determined, and an optimal transmission time sequence is calculated. Based on the principle of prioritizing high-priority data blocks, order data can be transmitted first, followed by inventory data, and finally user data. Based on the logical relationships between the different data blocks, an optimal transmission time sequence can be calculated to ensure data is transmitted in the most efficient manner.

[0039] S105. During the execution of the data migration task, continuously monitor the data synchronization quality and application system performance indicators to determine the smooth transition effect.

[0040] After enabling lossless data migration, source and target data are compared in real time to determine any inconsistencies or data loss between the two systems. The integrity of the migrated data is also verified to prevent omissions of fields or records. Data transmission, processing, and update delays between the source and target systems are measured, as well as the data integrity, consistency, and format errors during the data transfer process. Data synchronization quality is monitored in real time by recording the number of records or bytes transferred. Differences in the response time and throughput performance of business applications in the source and target systems are compared to determine the effectiveness of the smooth transition. This includes comparing the response time data of business applications in the source and target systems under the same load to determine the difference in response time. Comparing the throughput data of business applications in the source and target systems under the same load to determine the difference in throughput. Comparing the error rate data of business applications in the source and target systems under the same load to determine the difference in error rate. Comparing the availability of business applications in the source and target systems under the same load to determine the difference in availability.

[0041] For example, a data migration task is underway from source system A to target system B. During the migration process, the number of records transferred during each session is recorded, and data synchronization quality is monitored in real time. Real-time monitoring reveals the following data: the average data transfer delay from source system A to target system B is 5 seconds; the average data processing delay is 2 seconds; and the average data update delay is 3 seconds. During the data transfer process, the data integrity error rate is 0%, the data consistency error rate is 5%, and the data format error rate is 2%. Each transfer involves 1,000 records. In addition to monitoring data synchronization quality, performance metrics such as response time and throughput of business applications on source and target systems A and B can be compared to assess the effectiveness of the smooth transition. The response time data of business applications on source and target systems A and B under the same load can be compared. The average response time of source system A is 2 seconds, while that of target system B is 5 seconds. Furthermore, the throughput of source system A is 100 requests per second, while that of target system B is 120 requests per second. The error rate for source system A is 1 error per 100 requests, while the error rate for target system B is 5 errors per 100 requests. Finally, we can compare the availability of the business applications on source system A and target system B under the same load. The availability of source system A is 99%, while the availability of target system B is 95%.

[0042] S106. If a network bottleneck occurs during data migration, non-core services are suspended according to service priorities to release resources occupied by non-core services to ensure that the data of core services is migrated.

[0043] If during the migration process, it is detected that the core data migration link is congested and delayed, and the predetermined network performance index requirements cannot be met, then it is determined that a network bottleneck has occurred in the data migration. Among them, the services are divided into high-priority core services and low-priority non-core services based on their priorities. When a network bottleneck occurs in data migration, non-core services are suspended according to the priority of the services and the computing resources occupied by them are released. This includes: sorting the services according to the importance and urgency of the services and determining the priority of the services; obtaining the dependency relationships of each service and determining whether suspending low-priority non-core services will have an impact on the dependent services; if not, marking the service as suspendable; checking the resource occupancy details of each service and starting the backup resource pool; if starting the backup resource pool still cannot meet the predetermined network performance index requirements, suspending the services marked as suspendable and releasing the computing resources occupied by them.

[0044] For example, during a migration, the latency of the core data migration link was detected to be 100 milliseconds, while the predetermined network performance requirement was a latency of no more than 50 milliseconds. Based on service priorities, core services have an importance of 10 and an urgency of 5, while non-core services have an importance of 5 and an urgency of 3. Multiplying the priority by the urgency, the core service's priority is calculated to be 10 * 5 = 50, and the non-core service's priority is 5 * 3 = 15. After analyzing the dependencies of each service, it was found that suspending low-priority non-core services would not impact the core service. Therefore, the non-core services were marked as suspendable. A detailed breakdown of service resource usage revealed that core services occupied 80% of computing resources, while non-core services occupied 20%. After activating the backup resource pool, the remaining computing resources in the pool accounted for 60%. However, even with the backup resource pool activated, network performance still failed to meet the predetermined requirements. Therefore, based on service priorities, the non-core services marked as suspendable were suspended, freeing up their occupied computing resources.

[0045] S107: After the migration of core business data is completed, the suspended non-core business is gradually resumed until all business data in the standby data center are migrated and synchronized.

[0046] After waiting for the high-priority core business data migration to be completely completed, verify the progress monitoring and data integrity to confirm that the core business has been fully synchronized to the target data center. Obtain business associations, identify and gradually restore non-core businesses that have no dependencies on the core business. Restoring non-core businesses that have no dependencies on the core business includes: performing verification before restoring non-core businesses to ensure that no service data is missing or status errors occur during the suspension of non-core businesses; if there is any loss of incremental data, re-acquire the lost data from the source database. After restoring non-core businesses, verify business test cases to ensure that all businesses are completely switched and successfully deployed in the target data center, and that the source and target databases remain consistent.

[0047] For example, consider core business system A, which contains important customer order data. This order data needs to be migrated to target data center B. During the migration process, you need to wait for core business system A to complete the data migration, then monitor the progress and verify data integrity to ensure that target data center B is fully synchronized with source data center A. Core business system A has 1,000 orders to migrate, each containing data such as order numbers, customer information, and product information. You can use a progress monitoring tool to check the migration progress every 10 minutes. For example, 200 orders were migrated in the first 10 minutes, 400 orders in the 20th, and so on. You can calculate the migration progress percentage based on the number of migrated orders and the total number of orders. Furthermore, data integrity must be ensured. You can compare the order data in source database A and target database B by comparing fields such as order quantity, order number, and customer information to confirm data consistency. If inconsistencies are found in 10 order data during the comparison, these order data can be retrieved from the source database and re-migrated to the target database to ensure data integrity. Verification is also required during the recovery of non-core services that do not depend on core business system A. If a non-core business system C is dependent on core business system A, it is necessary to ensure that no service data is missing or status errors occur during the suspension of non-core business system C. You can confirm that no data is missing or errors occurred during the suspension by checking the logs or database of non-core business system C. After restoring non-core business system C, you need to verify business test cases to ensure that all services are completely switched over and successfully deployed in the target data center B, and that the source database A and the target database B remain consistent. You can run a series of test cases to check whether non-core business system C functions normally, such as user login and data query operations.

[0048] S108. Adopt a grayscale release model with gradually increased weight to smoothly transition the business to the data center, and establish a one-click automated fallback plan for the entire business environment to prevent abnormalities.

[0049] A phased rollout of business systems is performed gradually, from high to low priority, gradually increasing the weight of the new data center and smoothly transitioning services to the data center. The phased rollout continues without interrupting ongoing user sessions. Real-time business traffic is monitored, and if an anomaly is detected, an alert notification is issued. Upon receiving an alert notification, the anomaly is determined and business adjustments are made based on the result. A one-click, automated rollback solution for the entire business environment is established, allowing all services in the new data center to be quickly rolled back to the original production environment at once.

[0050] For example, business systems are released gradually in descending order of priority. The first batch of order systems has the highest priority, so they are selected for the grayscale release. Only 5% of production traffic is allowed to the first batch of order systems in the new data center to ensure that the order system's processing capacity does not exceed the target system's maximum capacity. A session persistence mechanism is enabled for users submitting orders to prevent errors when their network requests are redirected to the new system. Furthermore, the response time is capped at 50 milliseconds, and an alert is issued if this limit is exceeded. With a one-click automated rapid rollback solution in place, if the monitoring system indicates abnormal business metrics, a one-click automated full-service environment rollback is immediately initiated to quickly restore the order system batch to the original data center. Automated testing tools can be used to perform stress testing by sending large amounts of simulated business traffic to test all business function use cases and evaluate the effectiveness of the smooth migration. If the evaluation results are positive, the traffic weight of the new system can be further increased, and coverage can be gradually expanded.

[0051] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for dynamic data migration without perception, characterized in that: The method comprises: Obtain the differences between the source database and the target database and formulate a lossless data migration plan; compare the data differences between the source database and the target database, determine the amount of data that needs to be synchronized each time, and predict the required network bandwidth resources; if the current network bandwidth resources can meet the data migration needs, apply to expand the corresponding storage capacity and computing processing power as a backup resource pool for subsequent data migration; based on the importance and priority of the application system and data, use the Kalman filter algorithm to estimate the transmission time series and transmission path of different data blocks; continuously monitor the data synchronization quality and application system performance indicators during the execution of the data migration task to determine the effect of the smooth transition; if a network bottleneck occurs during data migration, suspend non-core businesses according to the priority of the business and release the resources occupied by non-core businesses to ensure that the core business data is migrated; wait for the core business data migration to be completed, and gradually resume the suspended non-core businesses until all business data migration and synchronization of the backup data center are completed; adopt a grayscale release model with gradually increasing weights to smoothly transition the business to the data center, and establish a one-click automated full business environment fallback plan to prevent abnormalities.

2. The method according to claim 1, wherein Obtaining the differences between the source and target databases and developing a lossless data migration plan includes: Obtain the differences between the table structures of the source and target databases, including differences in table names, column names, data types, primary keys, and indexes; If there are differences in the table structures between the source and target databases, a field mapping relationship is established to convert the target database table structure; Obtain incremental data change data between the source database and the target database, including: using a polling method to capture the add, delete, and modify operations of the source database in real time; converting the format of the incremental data and writing it to the target database based on the table structure and field mapping relationship between the source and target databases; formulating a data lossless migration plan, including: during the data migration process, continuously tracking the data differences between the source and target databases to determine whether there is any remaining data that needs to be migrated; if the business data of the source and target databases are associated, sort them according to the association and prioritize migrating the associated or dependent business data; if there are differences in the index structure between the source and target databases, the tables of the target database need to be re-indexed according to the index structure of the target database after the migration is completed; if there are differences in the constraints between the source and target databases, the constraints need to be gradually added to the tables of the target database after the migration is completed; if there are differences in the custom functions of the source and target databases, the custom functions of the target database need to be gradually added after the migration is completed.

3. The method according to claim 1, wherein The process of comparing the data differences between the source database and the target database, determining the amount of data to be synchronized each time, and predicting the required network bandwidth resources includes: Determine whether there are differences in the table structures between the source and target databases, and calculate the data volume of all tables to be migrated in the source database, including: if the table structures between the source and target databases are exactly the same, obtain the current data volume of the target database table; if there are differences in the table structures between the source and target databases, perform conversion calculations based on the field mapping relationship; calculate the amount of data that has been migrated historically; calculate the remaining amount of data to be migrated by subtracting the migrated amount from the total data volume; obtain the historical data change volume and predict the peak situation of synchronized data through a linear regression algorithm; calculate the required network bandwidth resources based on the network bandwidth calculation formula, combined with the peak data volume and the acceptable transmission time threshold.

4. The method according to claim 1, wherein If the current network bandwidth resources can meet the data migration requirements, then apply to expand the corresponding storage capacity and computing processing capabilities as a backup resource pool for subsequent data migration, including: Determine the backup capacity and resource redundancy based on the business availability requirements; use linear regression prediction algorithms to predict backup storage capacity requirements based on the business data volume and data growth rate; Establish a spare capacity assessment model based on spare capacity and resource redundancy and predicted spare storage capacity requirements; allocate spare storage capacity and computing resources based on the established spare capacity assessment model and the priority and importance of business systems; If it is assessed that the current network bandwidth resources can meet the expected data migration needs, then a data redundancy backup plan will be implemented to expand the capacity of the target data center based on the storage space requirements and computing processing power requirements after the subsequent migration is completed, including: expanding storage space and improving computing power based on the spare capacity assessment model; increasing resource monitoring and alarms; expanding storage space by adding physical machines or using virtualization components as a spare resource pool; computing power by vertically expanding existing physical machines or horizontally adding new servers; among which, increasing resource monitoring and alarms includes: determining whether there are faulty resources in the spare resource pool and recording the fault status; obtaining resource utilization history records through the spare resource pool management system; determining whether the resource utilization rate exceeds the preset threshold, and if so, triggering the corresponding alarm; determining the alarm notification method, and sending the alarm notification to relevant personnel.

5. The method according to claim 1, wherein The Kalman filter algorithm is used to estimate the transmission time sequence and transmission path of different data blocks according to the importance and priority of the application system and data, including: Determine the priority based on the importance of the business system. The system with the highest priority has the highest weight for data migration. The business database is divided into multiple data blocks, and each data block is marked in turn according to its importance, priority and business relevance; the candidate transmission paths of different data blocks are analyzed, and the candidate transmission paths of different data blocks are estimated using the Kalman filter algorithm; the logical relationship between different data blocks is determined, and the Kalman filter algorithm is used to integrate the data dependency relationship between different data blocks, and an optimal transmission time sequence is calculated based on the principle that high-priority data blocks are transmitted before low-priority data blocks and the logical relationship between different data blocks.

6. The method according to claim 1, wherein Continuously monitoring data synchronization quality and application system performance indicators during the data migration task to determine the smooth transition effect includes: After enabling lossless data migration, the system compares source and target data in real time to determine whether there are inconsistencies or data loss between the two. It also verifies the integrity of the migrated data to prevent omissions of fields or records. It also obtains the data transmission delay, data processing delay, and data update delay between the source and target systems, as well as the data integrity error rate, data consistency error rate, and data format error rate during the data transmission process. It also monitors data synchronization quality in real time by recording the number of records or bytes transferred. Compare the differences in response time and throughput performance indicators of business applications in the source center system and the target system to determine the effectiveness of the smooth transition. This includes: comparing the response time data of business applications in the source and target systems under the same load to determine the difference in response time; Compare the throughput data of the business applications of the source system and the target system under the same load to find the difference in throughput; compare the error rate data of the business applications of the source system and the target system under the same load to find the difference in error rate; compare the availability of the business applications of the source system and the target system under the same load to find the difference in availability.

7. The method according to claim 1, wherein If a network bottleneck occurs during data migration, non-core services will be suspended according to their priority, freeing up resources occupied by them to ensure the completion of data migration for core services, including: If during the migration process, it is detected that the core data migration link is congested and delayed, and the predetermined network performance index requirements cannot be met, then it is determined that a network bottleneck has occurred in the data migration; wherein, the services are divided into high-priority core services and low-priority non-core services based on priority; when a network bottleneck occurs in the data migration, the non-core services are suspended according to the service priority and the computing resources occupied by them are released, including: sorting the services according to the importance and urgency of the services and determining the service priority; obtaining the dependency relationship of each service and judging whether suspending the low-priority non-core services will affect the dependent services; if not, marking the service as suspendable; checking the resource occupancy details of each service and starting the backup resource pool; if starting the backup resource pool still cannot meet the predetermined network performance index requirements, suspending the services marked as suspendable and releasing the computing resources occupied by them.

8. The method according to claim 1, wherein After the core business data migration is completed, the suspended non-core business is gradually resumed until all business data in the standby data center is migrated and synchronized, including: After waiting for the high-priority core business data migration to be completely completed, verify the progress monitoring and data integrity to confirm that the core business has been fully synchronized to the target data center; obtain business associations, identify and gradually restore non-core businesses that have no dependencies on the core business; among them, restoring non-core businesses that have no dependencies on the core business includes: verifying before restoring non-core businesses to ensure that no service data is missing or status errors occur during the suspension of non-core businesses; if there is any loss of incremental data, retrieve the lost data from the source database; after restoring non-core businesses, verify business test cases to ensure that all businesses are completely switched and successfully deployed in the target data center, and that the source and target databases are consistent.

9. The method according to claim 1, wherein: The grayscale release model with gradually increasing weight is adopted to smoothly transition the business to the data center and establish a one-click automated full business environment fallback plan to prevent abnormalities, including: The business systems are released gradually in descending order of priority, gradually increasing the weight of the new data center and smoothly transitioning services to the data center. During the grayscale release, users' ongoing business operations are not interrupted. Real-time business traffic is monitored and warning notifications are issued if anomalies are detected. If an early warning notification is received, determine which link has an abnormality and make business adjustments based on the judgment result; establish a one-click automated full business environment rollback plan. The one-click automated full business environment rollback plan is to quickly roll back all businesses in the new data center to the original production environment at one time.

Images