Cross-platform database heterogeneous migration and fault-tolerant control method and system

Abstract

The application provides a cross-platform database heterogeneous migration and fault-tolerant control method and system, relates to the field of database migration, and comprises the following steps: constructing a heterogeneous mapping rule library; analyzing access logs to generate a hotness list with weights and calculating priorities in classification; establishing a task dependency topology graph and scheduling migration tasks; converting data according to rules and transmitting in slices; and synchronizing change information through a distributed message queue and checking consistency. The application can realize efficient migration between heterogeneous databases, reduce system downtime, ensure data consistency and improve migration efficiency.

Description

Technical Field

[0001] This invention relates to database migration technology, and more particularly to a cross-platform database heterogeneous migration and fault tolerance control method and system. Background Technology

[0002] With the rapid development of information systems, enterprises frequently face the need for cross-platform database migration. Traditional database migration methods mainly employ full export and import, which suffers from long migration cycles, prolonged business interruptions, and difficulties in ensuring data consistency. To address these issues, the industry has proposed online migration solutions that establish a real-time synchronization channel between the source and target databases to achieve incremental data migration.

[0003] In heterogeneous database environments, the log formats of different databases vary significantly, requiring the development of dedicated parsing modules for each database, and the log parsing process is prone to latency. While dual-write solutions, which implement data synchronization at the application layer and support heterogeneous database migration, are highly intrusive to application code and struggle to handle complex data structure transformations.

[0004] Furthermore, existing solutions generally lack analysis and optimization of data access characteristics. During large-scale data migration, the access frequency and business importance of different data vary significantly. If a uniform migration strategy is adopted, it is difficult to meet business continuity requirements. At the same time, in heterogeneous database environments, differences in data types and constraint rules also bring additional complexity to the migration process.

[0005] Therefore, a migration solution is needed that can support heterogeneous database platforms, ensure data consistency, and minimize business impact to meet the actual needs of enterprise database upgrades and transformations. Summary of the Invention

[0006] This invention provides a method and system for cross-platform database heterogeneous migration and fault tolerance control, which can solve the problems in the prior art.

[0007] A first aspect of this invention provides a method for cross-platform database heterogeneous migration and fault tolerance control, comprising:

[0008] Construct a heterogeneous mapping rule base from the source database to the target database;

[0009] Analyze the access logs of the source database, calculate the data access popularity score using a sliding time window, generate a weighted access popularity list, divide the data into hot datasets and cold datasets based on the access popularity list, and calculate the migration priority for each dataset.

[0010] A task dependency topology graph is established based on migration priority and heterogeneous mapping rule base. A migration task sequence is arranged through a parallel task scheduling algorithm to form a migration execution scheme that includes a data sharding strategy.

[0011] According to the migration execution plan, the data in the source database is converted into a new format according to the rules in the heterogeneous mapping rule base, and the converted data is then fragmented and transmitted to the target database.

[0012] During the transmission process, a distributed message queue is used to synchronize the source database change information to the target database in real time. Based on the heterogeneous mapping rule base, data consistency is ensured by performing incremental verification on the synchronized data and generating a consistency report. Based on the consistency report, an adaptive dual-write strategy is used to smoothly switch business traffic until the target database meets the performance requirements and completes the full migration.

[0013] In one alternative embodiment,

[0014] The heterogeneous mapping rule base from the source database to the target database includes:

[0015] Obtain data type information and data operation logs from the source database, and extract data distribution characteristics and field access records;

[0016] Based on the data distribution characteristics, dynamically calculate the numerical parameters required for the target storage and generate corresponding numerical conversion rules;

[0017] Based on the data distribution characteristics, a mapping association matrix between data structures is constructed. The composite structure of the source database is divided into transformation patterns using the mapping association matrix, and corresponding structure transformation rules are generated.

[0018] By analyzing the field access records, the dependencies between fields are analyzed, the constraints on field values ​​are extracted, and corresponding constraint transformation rules are generated.

[0019] The numerical transformation rules, structural transformation rules, and constraint transformation rules are integrated into a heterogeneous mapping rule library.

[0020] In one alternative embodiment,

[0021] Analyze the access logs of the source database, calculate the data access popularity score using a sliding time window, generate a weighted access popularity list, divide the data into hot datasets and cold datasets based on the access popularity list, and calculate the migration priority for each dataset, including:

[0022] Obtain access logs from the source database and extract access feature information and field dependencies for data items;

[0023] The initial sliding time window is determined based on the access feature information. Within the initial sliding time window, the access logs are grouped according to the transaction identifier to construct an access sequence diagram that reflects the access order of data items.

[0024] The access sequence diagram is analyzed to determine the access time relationship between data items. The propagation direction of data items is identified by combining the field dependency relationship. A hierarchical propagation structure of data items is established, and the node depth of each data item in the hierarchical propagation structure is calculated. The basic access weight of the data item is determined based on the node depth.

[0025] Within the initial sliding time window, the access frequency of data items is statistically analyzed, and the access importance of data items is calculated by combining the basic access weights.

[0026] The sliding time window size is dynamically adjusted based on the changing trend of access frequency. Within the updated sliding time window, a differentiated weight coefficient is determined based on the access importance of data items, and the access popularity score of data items is calculated and accumulated.

[0027] The access popularity scores are sorted to generate an access popularity list. The distribution characteristics of the access popularity list are analyzed, the abrupt change intervals of access popularity scores are identified and the boundary thresholds are determined. Based on the boundary thresholds, the data items are divided into hot datasets and cold datasets.

[0028] Migration priorities are calculated for hot and cold datasets based on access importance, access popularity score, and access interval, respectively.

[0029] In one alternative embodiment,

[0030] A task dependency topology graph is established based on migration priority and a heterogeneous mapping rule base. A migration task sequence is orchestrated using a parallel task scheduling algorithm, forming a migration execution scheme that includes a data sharding strategy.

[0031] The data items to be migrated are constructed as task nodes. The priority attribute of the task nodes is determined based on the migration priority, and the mapping attribute of the task nodes is determined based on the heterogeneous mapping rule base.

[0032] Analyze the dependencies between data items, establish directed connections between task nodes with dependencies, calculate the weights of directed connections based on the priority and mapping attributes of task nodes, and construct a task dependency topology graph.

[0033] Perform topology sorting on the task-dependent topology graph to generate a basic migration task sequence;

[0034] The mapping attributes of task nodes in the basic migration task sequence are analyzed. Task nodes with the same mapping attributes and no direct dependencies are grouped into parallel task groups. The data shard size is determined according to the mapping attributes of task nodes in the parallel task groups, and an initial data sharding strategy is generated.

[0035] The system resource status indicators are obtained to calculate the system processing capacity. The execution parallelism of the parallel task group is adjusted according to the system processing capacity. The execution parallelism is combined with the initial data sharding strategy to form a migration execution plan.

[0036] In one alternative embodiment,

[0037] According to the migration execution plan, the data in the source database is converted according to the rules in the heterogeneous mapping rule base, and the converted data is then fragmented and transmitted to the target database, including:

[0038] Transformation rules are extracted from a heterogeneous mapping rule base, the transformation rules are parsed into basic transformation units, the basic transformation units are compiled and optimized to generate independent transformation functions, and the complexity weight of each transformation function is calculated.

[0039] Adjust the data sharding strategy in the migration execution plan according to the complexity weight, and shard the data in the source database according to the adjusted sharding strategy;

[0040] Assign a unique identifier to each data fragment, construct a fragment transmission sequence, set processing priorities for each data fragment based on the complexity weight of the transformation function, and read data fragments from the source database in priority order.

[0041] Calculate the integrity characteristic value of the data fragment, allocate computing resources according to the processing priority of the data fragment, and call the corresponding conversion function to convert the data fragment into the target format;

[0042] The transformed data fragments and their integrity feature values ​​are transmitted to the target database, and the integrity of the received data is verified. Data fragments that fail verification are retransmitted.

[0043] In one alternative embodiment,

[0044] During transmission, a distributed message queue is used to synchronize changes from the source database to the target database in real time. Incremental verification is performed on the synchronized data, and a consistency report is generated, including:

[0045] A listener is deployed in the source database to obtain change information, which includes a transaction identifier, a timestamp, and the content of the changed data.

[0046] The change information is sorted according to the transaction identifier and timestamp, the change information associated with the transaction identifier is extracted, the dependency sequence between the change information is identified, the change information with dependency relationship is merged into atomic operation units, the heterogeneous mapping transformation is performed on the atomic operation units to generate change data in standard format, the real-time verification value and periodic verification value of the change data are calculated, and after being associated with the transaction identifier and timestamp, they are written into the verification record table.

[0047] Atomic operation units are sharded based on transaction identifiers and dependency sequences. Shards with dependencies are assigned to the same message queue, and the priority order and dependency conditions of the shards are recorded in the queue metadata.

[0048] Start multi-threaded consumption of sharded data in the message queue, verify the dependency conditions according to the queue metadata, perform synchronous writing for shards that meet the dependency conditions, and set a delay before re-enqueuing shards that do not meet the dependency conditions.

[0049] Consistency verification is performed on the synchronized shards based on real-time check values. At the same time, incremental verification is performed on the corresponding records in the target database based on periodic check values. When the incremental verification fails, the associated operations are traced based on the transaction identifier and timestamp to locate the source operation that caused the inconsistency.

[0050] Generate a consistency report by summarizing incremental verification results and anomaly location information.

[0051] In one alternative embodiment,

[0052] Based on consistency reports, an adaptive dual-write strategy is used to smoothly switch business traffic until the target database meets performance requirements and completes the full migration, including:

[0053] Obtain data synchronization delay information, abnormal operation distribution, and verification failure information from the consistency report, and collect performance metrics of the target database;

[0054] Based on the data synchronization delay information and performance indicators, a traffic switching threshold is determined, and a traffic adjustment strategy is formulated according to the abnormal operation distribution.

[0055] Monitor the performance metrics of the target database, and increase dual-write traffic when the traffic switching threshold is met; otherwise, adjust adaptively according to the traffic adjustment strategy.

[0056] The dual-write traffic is divided into read operation traffic and write operation traffic. The operation conflict level is determined based on the abnormal operation distribution, and the switching priority is divided according to the conflict level.

[0057] Monitor the target database cache status of read operation traffic, and switch write operation traffic according to transaction identifier and conflict level once the cache status meets the requirements.

[0058] Receive the incremental verification results from the consistency report. When data inconsistency occurs, extract the corresponding transaction identifier and adjust the corresponding dual-write traffic.

[0059] The migration progress is assessed based on the stability of the incremental verification results and the data synchronization delay information. The full migration is completed when the target database meets the performance requirements.

[0060] A second aspect of this invention provides a cross-platform database heterogeneous migration and fault-tolerant control system, comprising:

[0061] The first unit is used to build a heterogeneous mapping rule base from the source database to the target database;

[0062] The second unit is used to analyze the access logs of the source database, calculate the data access popularity score using a sliding time window, generate a weighted access popularity list, divide the data into hot datasets and cold datasets according to the access popularity list, and calculate the migration priority for each dataset.

[0063] The third unit is used to establish a task dependency topology graph based on migration priority and heterogeneous mapping rule base, and to arrange the migration task sequence through parallel task scheduling algorithm to form a migration execution scheme that includes data sharding strategy.

[0064] The fourth unit is used to convert the data in the source database according to the rules in the heterogeneous mapping rule base according to the migration execution plan, and then transmit the converted data in fragments to the target database.

[0065] The fifth unit is used to synchronize changes from the source database to the target database in real time using a distributed message queue during transmission. It also performs incremental verification on the synchronized data based on a heterogeneous mapping rule base to ensure data consistency and generates a consistency report. Based on the consistency report, it adopts an adaptive dual-write strategy to smoothly switch business traffic until the target database meets performance requirements and completes the full migration.

[0066] A third aspect of the present invention provides an electronic device, comprising:

[0067] processor;

[0068] Memory used to store processor-executable instructions;

[0069] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0070] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0071] In this embodiment, by constructing a heterogeneous mapping rule base and a data partitioning mechanism based on heat analysis, accurate mapping and priority scheduling are achieved during the cross-platform database migration process, improving migration efficiency and reducing access latency of business systems. The use of a task-dependent topology graph and parallel task scheduling algorithm, combined with a data sharding strategy, enables the migration process to have high parallelism and fault tolerance, effectively shortening migration time and automatically handling anomalies during migration, thus improving the stability of the migration process. Real-time synchronization of source database changes is achieved through a distributed message queue, combined with incremental verification and adaptive dual-write strategies, ensuring data consistency during the data migration process, achieving a smooth switchover of business systems, effectively avoiding the impact of the migration process on business continuity, and significantly reducing migration risks. Attached Figure Description

[0072] Figure 1 This is a flowchart illustrating the cross-platform database heterogeneous migration and fault tolerance control method according to an embodiment of the present invention;

[0073] Figure 2 This is a flowchart illustrating the heterogeneous database migration synchronization and adaptive dual-write traffic switching process in an embodiment of the present invention. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0076] Figure 1 This is a flowchart illustrating the cross-platform database heterogeneous migration and fault tolerance control method according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0077] Construct a heterogeneous mapping rule base from the source database to the target database;

[0078] Analyze the access logs of the source database, calculate the data access popularity score using a sliding time window, generate a weighted access popularity list, divide the data into hot datasets and cold datasets based on the access popularity list, and calculate the migration priority for each dataset.

[0079] A task dependency topology graph is established based on migration priority and heterogeneous mapping rule base. A migration task sequence is arranged through a parallel task scheduling algorithm to form a migration execution scheme that includes a data sharding strategy.

[0080] According to the migration execution plan, the data in the source database is converted into a new format according to the rules in the heterogeneous mapping rule base, and the converted data is then fragmented and transmitted to the target database.

[0081] During the transmission process, a distributed message queue is used to synchronize the source database change information to the target database in real time. Based on the heterogeneous mapping rule base, data consistency is ensured by performing incremental verification on the synchronized data and generating a consistency report. Based on the consistency report, an adaptive dual-write strategy is used to smoothly switch business traffic until the target database meets the performance requirements and completes the full migration.

[0082] In one alternative implementation, constructing a heterogeneous mapping rule base from the source database to the target database includes:

[0083] Obtain data type information and data operation logs from the source database, and extract data distribution characteristics and field access records;

[0084] Based on the data distribution characteristics, dynamically calculate the numerical parameters required for the target storage and generate corresponding numerical conversion rules;

[0085] Based on the data distribution characteristics, a mapping association matrix between data structures is constructed. The composite structure of the source database is divided into transformation patterns using the mapping association matrix, and corresponding structure transformation rules are generated.

[0086] By analyzing the field access records, the dependencies between fields are analyzed, the constraints on field values ​​are extracted, and corresponding constraint transformation rules are generated.

[0087] The numerical transformation rules, structural transformation rules, and constraint transformation rules are integrated into a heterogeneous mapping rule library.

[0088] In this implementation, during the stage of obtaining data type information and data operation logs from the source database, the metadata of the source database is accessed through a database connection interface to extract information such as table structure definitions, field types, and length limits. Simultaneously, a log collection component is deployed to collect database operation logs within a certain time window (e.g., 7 days), including SQL operation records such as SELECT, INSERT, UPDATE, and DELETE. For the user table of a certain e-commerce system, data type information such as user ID as an integer, username as a variable-length character type (maximum 50 characters), and registration time as a timestamp is extracted. Log analysis reveals that the actual maximum length of the username field is 20 characters, and 90% of query operations include the user ID field as a condition.

[0089] In the data distribution characteristic extraction stage, a full table scan or sampling analysis is performed on the source database, and statistics are calculated for each field. For numeric fields, the maximum, minimum, average, standard deviation, and quantiles are calculated; for character fields, the length distribution, the proportion of null values, and the frequency of special characters are calculated; for time-type fields, the time distribution range and granularity characteristics are analyzed. Taking the aforementioned user table as an example, the analysis shows that the user ID ranges from 1 to 10,000,000, showing a continuous distribution; the username length distribution is concentrated in the range of 8 to 15 characters and does not contain special characters; the registration time is concentrated in the weekday period from 9:00 to 18:00.

[0090] Field access record analysis extracts query patterns from collected operation logs, statistically analyzing the frequency and combination of each field in query, insert, and update operations. It records information such as the number of times each field is queried, the number of times it is used as a query condition, and the frequency of updates. For example, the user ID field in the user table appears as a condition in 95% of queries, the username appears in 30% of queries, while the user address field is used in only 5% of queries.

[0091] When dynamically calculating the numerical parameters required for target storage based on data distribution characteristics, conversion strategies are formulated for different data types. For integer data, the appropriate target type (such as TINYINT, INT, or BIGINT) is selected based on the actual value range; for floating-point numbers, the appropriate storage type is selected based on precision requirements; for strings, storage space is optimized based on the actual length distribution. In the user table conversion example, the user ID is selected as the INT type because its range is in the tens of millions; the username can be optimized from VARCHAR(50) to VARCHAR(25) based on the actual length distribution, saving storage space.

[0092] In the numeric conversion rule generation phase, a type mapping table is created from the source database to the target database, including conversion relationships for data types, lengths, and precision. For special types such as timestamps and enumerations, specific conversion functions are defined. For example, if the source database uses the DATETIME type to store time, while the target database uses TIMESTAMP, a corresponding conversion function is generated to ensure that time zone information is correctly preserved.

[0093] When constructing the data structure mapping association matrix, the relationships between tables in the source database are analyzed, including primary and foreign key relationships, referential integrity constraints, etc., and table join operations in the query log are used to generate a matrix of table association strength. For strongly related table groups, it is evaluated whether they are suitable for merging or splitting in the target database. For example, the user table and user details table are two separate tables in the source relational database, but in the target database (such as a document-oriented database), it may be more suitable to merge them into a nested structure.

[0094] The source database's complex structure is divided into transformation modes, and corresponding transformation templates are defined for different data structure characteristics (such as one-to-one, one-to-many, and many-to-many relationships). Analysis shows that the product and category tables in the source database have many-to-many relationships. In relational target databases, the intermediate table structure is maintained, while in document-oriented target databases, it is converted to array references or nested document structures.

[0095] When generating structure transformation rules, detailed transformation instructions are created for each structure type based on the defined transformation patterns, including table creation, field mapping, and index design. For the order table and its order item details, rules are generated to transform one-to-many relationships into nested document structures, while retaining appropriate indexes to support high-frequency query patterns. When analyzing field dependencies using field access records, a field co-occurrence matrix is ​​constructed to identify highly related field groups. By analyzing query patterns, it was found that the price and discount fields in the product table are always accessed together, indicating a close business relationship between them. The constraints on field values ​​are extracted, and the CHECK constraints, triggers, and application-layer validation logic in the source database are analyzed to extract business rules. Examples include rules such as user status fields only accepting "active," "inactive," and "suspended" values, and product prices must be greater than zero.

[0096] During the constraint transformation rule generation process, the extracted constraints are converted into constraint forms supported by the target database, such as CHECK constraints, foreign key constraints, or triggers. For constraint types not directly supported by the target database, the system generates alternative implementations, such as using triggers to simulate CHECK constraints. Finally, the generated numerical transformation rules, structural transformation rules, and constraint transformation rules are integrated into a unified heterogeneous mapping rule library, stored as a JSON or XML configuration file containing complete mapping definitions and transformation instructions for use by data migration tools. The rule library supports version control and incremental updates to adapt to changes in the source database structure.

[0097] In this embodiment, by extracting data distribution characteristics and field access records, numerical conversion rules, structural conversion rules, and constraint conversion rules can be dynamically generated, thereby ensuring accurate mapping of data types, composite structures, and field constraints during the migration process. Using a mapping association matrix to classify the conversion patterns of composite structures can effectively handle complex data structures and improve the integrity and consistency of the migration. Simultaneously, the analysis of field dependencies and value constraints ensures the preservation and correctness of data constraints in the target database, thereby reducing the risk of data anomalies after migration and achieving high reliability, scalability, and automation in data migration.

[0098] In one optional implementation, the access logs of the source database are analyzed, a sliding time window is used to calculate the data access popularity score, a weighted access popularity list is generated, the data is divided into hot datasets and cold datasets according to the access popularity list, and migration priority is calculated for each dataset, including:

[0099] Obtain access logs from the source database and extract access feature information and field dependencies for data items;

[0100] The initial sliding time window is determined based on the access feature information. Within the initial sliding time window, the access logs are grouped according to the transaction identifier to construct an access sequence diagram that reflects the access order of data items.

[0101] The access sequence diagram is analyzed to determine the access time relationship between data items. The propagation direction of data items is identified by combining the field dependency relationship. A hierarchical propagation structure of data items is established, and the node depth of each data item in the hierarchical propagation structure is calculated. The basic access weight of the data item is determined based on the node depth.

[0102] Within the initial sliding time window, the access frequency of data items is statistically analyzed, and the access importance of data items is calculated by combining the basic access weights.

[0103] The sliding time window size is dynamically adjusted based on the changing trend of access frequency. Within the updated sliding time window, a differentiated weight coefficient is determined based on the access importance of data items, and the access popularity score of data items is calculated and accumulated.

[0104] The access popularity scores are sorted to generate an access popularity list. The distribution characteristics of the access popularity list are analyzed, the abrupt change intervals of access popularity scores are identified and the boundary thresholds are determined. Based on the boundary thresholds, the data items are divided into hot datasets and cold datasets.

[0105] Migration priorities are calculated for hot and cold datasets based on access importance, access popularity score, and access interval, respectively.

[0106] In practical applications, obtaining the access logs of the source database is the first crucial step. Access logs typically contain information such as transaction IDs, operation timestamps, SQL statements, accessed data item identifiers, and operation types. The access characteristic information of data items is extracted from the logs, including access frequency, access time intervals, and access patterns. Simultaneously, JOIN operations and foreign key constraints in the SQL statements are analyzed to establish a dependency graph between fields. For example, in an e-commerce system, analyzing the access logs reveals a dependency between the `order_id` field in the `order_id` table and the `order_id` field in the `order_details` table, and a dependency between the `customer_id` field in the `customer_id` table and the `customer_id` field in the `order_id` table.

[0107] When determining the initial sliding time window, it should be set according to business characteristics and data access patterns. For transaction-intensive applications, the initial window can be set to 4 hours; for batch processing applications, it can be set to 24 hours. Within this window, access logs are grouped according to transaction identifiers to construct an access sequence graph. For example, if transaction T1 accesses tables A, B, and C in sequence, and transaction T2 accesses tables B and D in sequence, a directed graph can be constructed based on this, with nodes representing data tables and edges representing the access order.

[0108] When analyzing the access sequence relationships between data items based on access sequence diagrams, frequently occurring access paths are identified, such as the A→B→C path appearing 100 times and the B→D path appearing 80 times. Combined with field dependencies, the direction of data propagation is further identified. For example, changes to the customer table will trigger accesses to the order table and order details table, rather than the other way around. Through this analysis, a hierarchical propagation structure of data items is established, and the node depth of each data item in the structure is calculated. For example, the customer table is at the top level with a depth of 1; the order table is at the second level with a depth of 2; and the order details table is at the third level with a depth of 3. The smaller the depth, the earlier the data item is in the business process, the wider its influence, and the higher its basic access weight. Specifically, the basic access weight can be set as the reciprocal of the depth; for example, a depth of 1 has a weight of 1.0, a depth of 2 has a weight of 0.5, and a depth of 3 has a weight of 0.33.

[0109] When analyzing the access frequency of data items within the initial sliding time window, record the number of times each data item is accessed. For example, the customer table might be accessed 1000 times, the order table 3000 times, and the order details table 5000 times. Combine this with a base access weight to calculate the access importance of each data item. Access importance equals access frequency multiplied by the base access weight. For instance, the access importance of the customer table is 1000 × 1.0 = 1000, the access importance of the order table is 3000 × 0.5 = 1500, and the access importance of the order details table is 5000 × 0.33 = 1650.

[0110] The sliding window size is dynamically adjusted based on the trend of access frequency. If the recent access frequency shows an upward trend, the window size can be reduced by 25%; if it shows a downward trend, the window can be expanded by 25%. Within the updated window, a differentiated weighting coefficient is determined based on the access importance of the data items. For data items ranking in the top 20% of access importance, the weighting coefficient is set to 1.5; for those ranking 20%-50%, it is set to 1.2; for those ranking 50%-80%, it is set to 1.0; and for those ranking in the bottom 20%, it is set to 0.8. The access frequency is multiplied by the basic access weight and the differentiated weighting coefficient to calculate and accumulate the access popularity score of the updated data items. For example, within the new window, the access popularity score of the customer table is 1200 × 1.0 × 1.5 = 1800, the access popularity score of the order table is 3500 × 0.5 × 1.2 = 2100, and the access popularity score of the order details table is 4800 × 0.33 × 1.0 = 1584.

[0111] After sorting the access popularity scores to generate an access popularity list, their distribution characteristics are analyzed, and clustering algorithms are used to identify abrupt changes in the access popularity scores. For example, if the sorted access popularity scores are [5000, 4800, 4200, 3900, 2100, 1800, 1584, 800, 750, 600...], a significant gap is detected between 2100 and 800, and this gap is set as the dividing threshold of 1500. Based on this threshold, the data items are divided into hot datasets (score ≥ 1500) and cold datasets (score < 1500).

[0112] For hot and cold datasets, migration priorities are calculated separately. The migration priority for hot datasets primarily considers access popularity score and access importance, calculated as: popularity score multiplied by access importance divided by 100. For example, the migration priority for an order table is 2100 × 1500 ÷ 100 = 31500. The migration priority for cold datasets primarily considers access interval and access importance, calculated as: the reciprocal of the number of days since the most recent access multiplied by access importance. For example, if a cold data table was last accessed 10 days ago and has an access importance of 600, then its migration priority is 0.1 × 600 = 60.

[0113] The above methods enable intelligent hierarchical management and optimization of the database migration process. By analyzing access logs and field dependencies, access patterns and propagation characteristics of data items can be dynamically identified, a hierarchical propagation structure for data items can be constructed, and access importance can be calculated based on node depth and access frequency, thus accurately reflecting the data's usage value. Combined with sliding time windows and weight adjustments, changes in access popularity can be tracked in real time, enabling dynamic calculation of access popularity scores. Hot and cold data are identified using abrupt changes in access popularity, and migration priorities are generated based on access frequency and intervals, ensuring that hot data is migrated first and cold data is migrated later. This solution improves the efficiency and resource utilization of data migration while reducing migration risks, achieving intelligent, differentiated, and predictable migration processes.

[0114] In one optional implementation, a task dependency topology graph is established based on migration priority and a heterogeneous mapping rule base. A migration task sequence is orchestrated using a parallel task scheduling algorithm to form a migration execution scheme that includes a data sharding strategy.

[0115] The data items to be migrated are constructed as task nodes. The priority attribute of the task nodes is determined based on the migration priority, and the mapping attribute of the task nodes is determined based on the heterogeneous mapping rule base.

[0116] Analyze the dependencies between data items, establish directed connections between task nodes with dependencies, calculate the weights of directed connections based on the priority and mapping attributes of task nodes, and construct a task dependency topology graph.

[0117] Perform topology sorting on the task-dependent topology graph to generate a basic migration task sequence;

[0118] The mapping attributes of task nodes in the basic migration task sequence are analyzed. Task nodes with the same mapping attributes and no direct dependencies are grouped into parallel task groups. The data shard size is determined according to the mapping attributes of task nodes in the parallel task groups, and an initial data sharding strategy is generated.

[0119] The system resource status indicators are obtained to calculate the system processing capacity. The execution parallelism of the parallel task group is adjusted according to the system processing capacity. The execution parallelism is combined with the initial data sharding strategy to form a migration execution plan.

[0120] In this implementation, the data items to be migrated are first constructed as task nodes. For a database migration scenario containing multiple tables, each table is treated as a data item and constructed as a task node. For example, in an e-commerce system, each table—including user tables, order tables, product tables, payment record tables, and logistics information tables—is constructed as a separate task node. Task nodes include basic attributes such as node identifier, data item name, data size, and data structure information.

[0121] The priority attributes of task nodes are determined based on migration priority. In this implementation, the priority is divided into five levels: critical (5), high (4), medium (3), low (2), and deferred (1). The priority determination criteria include business importance, access frequency, and data update rate. For example, the user table is set to critical (5), the order table to high (4), the product table to medium (3), the historical transaction record table to low (2), and the log table to deferred (1).

[0122] The mapping attributes of task nodes are determined based on a heterogeneous mapping rule base. This base stores rules for numerical conversion, structural conversion, and constraint conversion between different database systems. For example, it maps VARCHAR data types in MySQL to String data types in MongoDB, and auto-incrementing primary keys in MySQL to ObjectId data types in MongoDB. By querying the heterogeneous mapping rule base, corresponding mapping attributes are assigned to each task node, including the target system data type, conversion method identifier, and complexity evaluation value.

[0123] Analyze the dependencies between data items and establish directed connections between task nodes. Identify dependencies between data items by analyzing foreign key constraints, trigger relationships, and business logic associations between tables. For example, the order table depends on the user table (the user ID in the order table references the primary key in the user table), and the product review table depends on both the product table and the user table. For each pair of task nodes with dependencies, establish a directed connection from the dependent node to the dependent node.

[0124] The weight of a directed join is calculated based on the priority and mapping attributes of the task nodes. The join weight is calculated by comprehensively considering the source node priority, the target node priority, and the mapping complexity. For example, when the source node is a user table (priority 5), the target node is an order table (priority 4), and the mapping complexity is medium (value 3), the join weight is calculated as 35 (through a weighted combination of priority and complexity). A higher weight value indicates a more critical dependency.

[0125] Construct a task dependency topology graph. Combine all task nodes and directed connections to form a directed weighted graph structure. In this graph, nodes represent migration tasks, edges represent dependencies, and edge weights represent dependency strengths. For example, in a migration scenario involving 5 tables, the constructed topology graph contains 5 nodes and 7 directed connections, forming a complete dependency network.

[0126] Perform topological sorting on the task-dependent topology graph to generate a basic migration task sequence. The Kahn topological sorting algorithm is used, considering connection weights. The Kahn algorithm is an algorithm for sorting vertices in directed acyclic graphs (DAGs), primarily used to determine task execution order or handle dependencies. First, all nodes with an in-degree of 0 are identified and added to a candidate queue, sorted by node priority. The highest-priority node is then removed from the candidate queue and added to the result sequence, with all its outgoing edges removed and the in-degree of the relevant nodes updated. This process is repeated until all nodes are added to the result sequence. For example, the sorted result might be: User Table → Product Table → Order Table → Payment Record Table → Logistics Information Table.

[0127] Analyze the mapping attributes of task nodes in the basic migration task sequence and group them into parallel task groups. Examine the nodes in the task sequence and group nodes with the same mapping attributes and no direct dependencies into parallel task groups. For example, the product table and the brand table have the same mapping attributes and no direct dependencies, so they can be grouped into a parallel task group. Nodes with different mapping attributes are executed separately.

[0128] The data shard size is determined based on the mapping attributes of the task nodes in the parallel task group, generating an initial data sharding strategy. For each parallel task group, a suitable shard size is calculated based on its mapping complexity and data volume. For example, for a large table (5 million records) with a simple mapping (complexity 1), each shard can be set to contain 100,000 records; for a medium-sized table (1 million records) with a complex mapping (complexity 4), each shard can be set to contain 20,000 records.

[0129] Obtain system resource status indicators to calculate system processing capacity. Monitor system resource indicators such as CPU utilization, memory usage, network bandwidth, and disk I / O to comprehensively evaluate the current system's processing capacity. For example, when CPU utilization is 60%, available memory is 8GB, and network bandwidth is 100MB / s, the system's processing capacity is assessed as medium, capable of supporting 3-5 parallel tasks.

[0130] Adjust the execution parallelism of parallel task groups based on system processing capacity. Dynamically adjust the execution parallelism of parallel task groups based on system processing capacity and task complexity. For example, for a system with medium processing capacity, set low-complexity parallel task groups to 5 parallel instances and high-complexity parallel task groups to 3 parallel instances.

[0131] The execution parallelism is combined with the initial data sharding strategy to form a migration execution plan. This plan integrates parallel task groups, execution parallelism, and data sharding strategies to generate a complete migration execution plan. This plan includes details such as task execution order, task grouping information, parallelism settings for each task group, and data sharding strategy. For example, for a parallel task group consisting of a product table and a brand table, five parallel instances are set up, each processing a data shard of 20,000 records, forming a specific execution unit. The final generated migration execution plan serves as input to the data migration engine, guiding the sequential execution and resource allocation throughout the migration process.

[0132] In this embodiment, a task dependency topology graph is constructed to clarify the priority and mapping relationships between data items, enabling scientific sorting and dependency management of migration tasks. A parallel task scheduling algorithm can group independent task nodes with the same mapping attributes into parallel task groups, and combined with a data sharding strategy, improve the parallelism and throughput of the migration. The execution degree of parallel tasks is dynamically adjusted according to the system's processing capacity, ensuring that the migration process fully utilizes computing resources and avoids resource bottlenecks and task blocking. This scheme significantly improves migration efficiency and resource utilization while ensuring the correctness of data dependencies and mapping rules, achieving intelligent, parallel, and controllable database migration.

[0133] In one optional implementation, according to the migration execution plan, the data in the source database is formatted according to the rules in the heterogeneous mapping rule base, and the converted data is fragmented and transmitted to the target database, including:

[0134] Transformation rules are extracted from a heterogeneous mapping rule base, the transformation rules are parsed into basic transformation units, the basic transformation units are compiled and optimized to generate independent transformation functions, and the complexity weight of each transformation function is calculated.

[0135] Adjust the data sharding strategy in the migration execution plan according to the complexity weight, and shard the data in the source database according to the adjusted sharding strategy;

[0136] Assign a unique identifier to each data fragment, construct a fragment transmission sequence, set processing priorities for each data fragment based on the complexity weight of the transformation function, and read data fragments from the source database in priority order.

[0137] Calculate the integrity characteristic value of the data fragment, allocate computing resources according to the processing priority of the data fragment, and call the corresponding conversion function to convert the data fragment into the target format;

[0138] The transformed data fragments and their integrity feature values ​​are transmitted to the target database, and the integrity of the received data is verified. Data fragments that fail verification are retransmitted.

[0139] This implementation first extracts conversion rules applicable to the current migration task from the heterogeneous mapping rule base. For example, when migrating a customer table from a relational database to a document database, the rules that may be extracted include: "convert integer ID to string type", "convert DATE type to ISO8601 format string", "convert inter-table relationships to embedded documents", etc.

[0140] After extracting the rules, these rules are parsed into basic transformation units. A basic transformation unit is the smallest unit for performing a specific transformation operation, such as "INT_TO_STRING" or "DATE_FORMAT_CONVERT". For a client representation example, the parsed basic transformation units might include: "CUSTOMER_ID_INT_TO_STRING", "BIRTH_DATE_TO_ISO8601", "ADDRESS_RELATION_TO_EMBEDDED", etc.

[0141] For each basic conversion unit, an independent conversion function is generated through compilation optimization techniques. The optimization process includes eliminating redundant operations, merging similar conversions, and inlining simple functions to improve execution efficiency. For example, for multiple fields requiring date format conversion, a unified date conversion function can be generated instead of generating a separate function for each field. The generated functions are stored in code form, such as "convertCustomerId(int id)" and "convertBirthDate(Date date)".

[0142] Calculate the complexity weight for each transformation function, considering factors such as computational complexity, memory usage, and I / O operations. The complexity weight is represented numerically; for example, a simple integer-to-string conversion might have a weight of 1, while a complex relational reconstruction might have a weight of 10. For the client representation example, "CUSTOMER_ID_INT_TO_STRING" might have a weight of 1, while "ADDRESS_RELATION_TO_EMBEDDED" might have a weight of 8, because the latter requires querying the related table and reconstructing the data structure.

[0143] Based on the complexity weights of the transformation functions, the data sharding strategy in the migration execution plan is adjusted. More fine-grained sharding is used for high-complexity transformation regions, while larger shards can be used for low-complexity regions. For example, records in a customer table containing address associations might be divided into smaller shards (500 records per shard), while records containing only basic information might use larger shards (5000 records per shard). After adjusting the sharding strategy, the data in the source database is sharded according to the strategy. The sharding process considers the logical integrity of the data, ensuring that closely related data is placed in the same shard as much as possible. For example, a customer table with 100,000 records might be divided into 30 shards, where high-complexity data regions (such as customer records containing a large number of associated addresses) are divided into 20 smaller shards, while simpler data regions are divided into 10 larger shards.

[0144] Assign a globally unique identifier (GUID) to each data fragment, such as "CUSTOMER_SHARD_001_20230815120000", to ensure accurate tracking of each fragment throughout the migration process. Based on fragment dependencies and the complexity weights of the transformation functions, a fragment transmission sequence is constructed to determine the order in which fragments are processed. For example, for a client representation, fragments containing complex address associations might be prioritized high, basic information fragments medium, and fragments without associated information low. Data fragment content is then read from the source database according to priority.

[0145] After reading the data fragments, the integrity feature value of each fragment is calculated, such as using the SHA-256 hash algorithm to calculate the hash value of the data content. For example, the feature value of the fragment "CUSTOMER_SHARD_001" might be "7f83b1657ff1fc53b92dc18148a1d65dfc2d4b1fa3d677284addd200126d9069". Computational resources are allocated based on the processing priority of the data fragments. Higher-priority fragments may receive more CPU threads and memory resources. For example, a fragment processing complex address associations might be allocated 8 CPU threads, while a fragment processing basic information might only be allocated 2 threads.

[0146] The corresponding conversion functions are called to convert the data shards to the target format. Detailed logs are recorded during the conversion process, including the start and end times, the number of records processed, and any exceptions. For example, for the shard "CUSTOMER_SHARD_001" in the customer table, functions such as "convertCustomerId" and "convertBirthDate" are called to convert the data.

[0147] After the conversion is complete, the converted data fragments and their integrity feature values ​​are transmitted to the target database. The transmission uses an encrypted channel to ensure data security. Upon receiving the data, the target database recalculates the data's feature values ​​and compares them with the transmitted feature values ​​to verify data integrity. If verification fails, a retransmission mechanism is automatically triggered. During retransmission, fragment sizes or transmission parameters may be adjusted to improve the success rate. For example, if fragment "CUSTOMER_SHARD_001" fails verification, it may be further divided into two sub-fragments and retransmitted until verification succeeds. Through this series of operations, efficient and reliable heterogeneous database migration is achieved.

[0148] Based on the above technical solutions, efficient, reliable, and intelligent data migration from the source database to the target database can be achieved. By parsing heterogeneous mapping rules into basic transformation units and generating optimized transformation functions, precise format conversion can be performed for different data types and structures. Simultaneously, the data sharding strategy is adjusted according to the transformation complexity, improving migration parallelism and resource utilization efficiency. Assigning unique identifiers and priorities to data shards, and combining them with integrity feature values ​​for transmission and verification, ensures data integrity and consistency during the migration process. Retransmission is supported for data shards that fail verification, further reducing migration risks and achieving controllability, automation, and high reliability in the migration process.

[0149] like Figure 2 As shown, this embodiment demonstrates the heterogeneous database migration synchronization and adaptive dual-write traffic switching process.

[0150] In one optional implementation, during transmission, a distributed message queue is used to synchronize changes from the source database to the target database in real time. Incremental verification is performed on the synchronized data, and a consistency report is generated, including:

[0151] A listener is deployed in the source database to obtain change information, which includes a transaction identifier, a timestamp, and the content of the changed data.

[0152] The change information is sorted according to the transaction identifier and timestamp, the change information associated with the transaction identifier is extracted, the dependency sequence between the change information is identified, the change information with dependency relationship is merged into atomic operation units, the heterogeneous mapping transformation is performed on the atomic operation units to generate change data in standard format, the real-time verification value and periodic verification value of the change data are calculated, and after being associated with the transaction identifier and timestamp, they are written into the verification record table.

[0153] Atomic operation units are sharded based on transaction identifiers and dependency sequences. Shards with dependencies are assigned to the same message queue, and the priority order and dependency conditions of the shards are recorded in the queue metadata.

[0154] Start multi-threaded consumption of sharded data in the message queue, verify the dependency conditions according to the queue metadata, perform synchronous writing for shards that meet the dependency conditions, and set a delay before re-enqueuing shards that do not meet the dependency conditions.

[0155] Consistency verification is performed on the synchronized shards based on real-time check values. At the same time, incremental verification is performed on the corresponding records in the target database based on periodic check values. When the incremental verification fails, the associated operations are traced based on the transaction identifier and timestamp to locate the source operation that caused the inconsistency.

[0156] Generate a consistency report by summarizing incremental verification results and anomaly location information.

[0157] In practical applications, the source database can be a relational database such as MySQL, and the target database can be a non-relational database such as MongoDB. First, a listener is deployed on the source database. This listener is implemented based on the database's binary log mechanism. Taking MySQL as an example, the listener obtains data change information by parsing the binlog. The listener is configured to read events from the binlog and extract transaction identifiers (such as XID), timestamps (recording the precise time the change occurred, accurate to milliseconds), and changed data content (including table name, column name, operation type, value before change, and value after change). For example, when a user inserts a record {id: 10001, name: "Zhang San", age: 30} into the customer table of the source database, the listener captures the event, extracts the transaction identifier as "TX-20240501-123456", the timestamp as "2024-05-01 10:15:32.456", and the changed content as "INSERT INTO customer(id, name, age) VALUES(10001, 'Zhang San', 30)".

[0158] After obtaining the change information, it is sorted according to transaction identifier and timestamp. The sorting process uses a multi-way merge sort algorithm to ensure that operations within a transaction are arranged in the original execution order. By analyzing the table and field relationships in the change information, the dependency sequence between change information is identified. For example, when a foreign key in table A references the primary key in table B, the insert operation in table B will be prioritized over the insert operation in table A. For identified related transactions, they are merged into atomic operation units. For example, in an order system, creating an order and updating the inventory are merged into a single atomic unit.

[0159] When performing heterogeneous mapping transformation on atomic operation units, maintain a mapping configuration table to record mapping relationships such as field types and naming rules between the source database and the target database. For example, the VARCHAR type in MySQL is mapped to the String type in MongoDB, and the integer type INT is mapped to the Number type. By applying these mapping rules, generate change data in a standard format, such as JSON format. Example of change data: {"operation": "INSERT", "table": "customer", "data": {"id": 10001, "name": "Zhang San", "age": 30}, "transaction_id": "TX-20240501-123456", "timestamp": "2024-05-01 10:15:32.456"}.

[0160] Calculate two types of check values for each piece of change data: The real-time check value is calculated using the CRC32 algorithm for the key fields of the change data; the periodic check value is calculated using the SHA-256 algorithm for the complete record content. These check values are written into the check record table together with the transaction identifier and timestamp. The table structure includes fields: record_id, transaction_id, timestamp, table_name, primary_key, real_time_checksum, periodic_checksum, sync_status.

[0161] When sharding atomic operation units based on the transaction identifier and dependency sequence, the system uses the consistent hashing algorithm to ensure that operations with the same primary key are assigned to the same shard. The shard size is dynamically adjusted according to the data volume, and the default setting is 500KB. For shards with dependencies, assign them to the same message queue. In a distributed message queue (such as Kafka), create multiple topics, and each topic contains multiple partitions. Record the priority order and dependency conditions of the shards in the queue metadata. The metadata structure includes: slice_id, priority, dependencies, status. For example, the metadata of a certain shard is {"slice_id": "S-10001", "priority": 2, "dependencies": ["S-10000"], "status": "pending"}, indicating that this shard depends on the completion of the execution of shard S-10000.

[0162] A multi-threaded consumer is launched (the number of threads is dynamically adjusted based on system resources, defaulting to twice the number of processor cores) to read sharded data from the message queue. Before processing a shard, each consumer thread checks the dependency conditions based on the queue's metadata. For example, for the shard S-10001 that depends on S-10000, the system checks the execution status of S-10000. If the dependency condition is met (S-10000 has completed execution), synchronous writing is performed immediately; otherwise, a delay parameter is set (using an exponential backoff strategy, with an initial delay of 100 milliseconds and a maximum delay of 10 seconds) before re-enqueuing. During synchronous writing, the system uses the target database's batch write API to improve efficiency, such as MongoDB's `bulkWrite` operation.

[0163] After synchronization is complete, consistency verification is performed on the synchronized shards based on real-time checksums. This verification process involves recalculating the checksums of records in the target database and comparing them with the checksums of the source records. Simultaneously, incremental checks are performed based on periodic checksums, occurring every 10 minutes, randomly selecting the most recently synchronized 10% of data for a full comparison. When inconsistencies are detected, the associated operations are traced based on the transaction identifier and timestamp, querying historical records in the checksum table to locate the source operation causing the inconsistency. For example, if the record with customer ID 10001 has an age value of 31 in the target database, which is inconsistent with the source database, tracing back reveals an incorrectly synchronized update operation within transaction "TX-20240501-123789".

[0164] Finally, the incremental verification results and anomaly location information are summarized to generate a consistency report. The report includes: the total number of synchronized records, success rate, list of failed records, details of inconsistent records (including table name, primary key, inconsistent fields, source value, and target value), and transaction information for the abnormal operations. The report is stored in JSON format and can be viewed in real time via a web interface. An alarm mechanism is also provided; when the inconsistency rate exceeds a preset threshold (default 0.1%), a notification is sent to the administrator via email or push notification.

[0165] Based on the above technical solutions, real-time, reliable synchronization and consistency assurance of source database change information can be achieved. By monitoring source database changes and merging them into atomic operation units according to transaction dependencies, the order and dependencies of data operations can be precisely maintained. Distributed message queues are used for sharded transmission and multi-threaded consumption, improving synchronization efficiency and ensuring the correct execution of dependency conditions. Incremental verification of synchronized data is performed by combining real-time and periodic checksums, enabling timely detection of data inconsistencies and precise location of source operation anomalies. Finally, a consistency report is generated, achieving high reliability, traceability, and data consistency assurance during database migration or synchronization.

[0166] In one optional implementation, an adaptive dual-write strategy based on consistency reports is used to smoothly switch business traffic until the target database meets performance requirements and completes the full migration, including:

[0167] Obtain data synchronization delay information, abnormal operation distribution, and verification failure information from the consistency report, and collect performance metrics of the target database;

[0168] Based on the data synchronization delay information and performance indicators, a traffic switching threshold is determined, and a traffic adjustment strategy is formulated according to the abnormal operation distribution.

[0169] Monitor the performance metrics of the target database, and increase dual-write traffic when the traffic switching threshold is met; otherwise, adjust adaptively according to the traffic adjustment strategy.

[0170] The dual-write traffic is divided into read operation traffic and write operation traffic. The operation conflict level is determined based on the abnormal operation distribution, and the switching priority is divided according to the conflict level.

[0171] Monitor the target database cache status of read operation traffic, and switch write operation traffic according to transaction identifier and conflict level once the cache status meets the requirements.

[0172] Receive the incremental verification results from the consistency report. When data inconsistency occurs, extract the corresponding transaction identifier and adjust the corresponding dual-write traffic.

[0173] The migration progress is assessed based on the stability of the incremental verification results and the data synchronization delay information. The full migration is completed when the target database meets the performance requirements.

[0174] For example, during the data migration process, key information from the consistency report is first obtained. The consistency report is generated by comparing the source and target databases and includes data synchronization latency information, abnormal operation distribution, and verification failure information. Data synchronization latency information records the synchronization latency of each data table; for example, the synchronization latency for the user table is 15 milliseconds, the order table is 25 milliseconds, and the product table is 18 milliseconds. Abnormal operation distribution records the abnormality rate of various operations; for example, the abnormality rate for read operations is 0.5%, and the abnormality rate for write operations is 1.2%, with the abnormality rate for order update operations reaching 2.3%. Verification failure information records the specific tables and fields where data is inconsistent; for example, five records in the order table have inconsistent order status fields. Simultaneously, performance metrics of the target database are collected, including CPU utilization, memory utilization, I / O throughput, and query response time; for example, the current CPU utilization is 45%, and the average query response time is 8 milliseconds.

[0175] Based on the collected information, traffic switching thresholds and traffic adjustment strategies are determined. Traffic switching thresholds are determined according to data synchronization latency and target database performance metrics; for example, increased traffic is allowed when synchronization latency is less than 30 milliseconds and CPU utilization is below 70%. Traffic adjustment strategies are formulated based on the distribution of abnormal operations, adopting a more conservative switching strategy for operation types with high abnormality rates. For example, for order update operations with an abnormality rate of 2.3%, only 5% of traffic is increased each time; while for read operations with an abnormality rate of only 0.5%, 15% of traffic can be increased each time.

[0176] During traffic switching, continuously monitor the target database's performance metrics. When the target database's CPU utilization reaches 62%, memory utilization reaches 58%, and query response time is 12 milliseconds, the traffic switching threshold is met, and the dual-write traffic ratio is increased, for example, from the current 30% to 45%. If the CPU utilization suddenly increases to 85%, exceeding the threshold, adaptively adjust according to the traffic adjustment strategy, rolling back the traffic to 35%, and extending the interval between the next traffic increase from the original 10 minutes to 20 minutes.

[0177] Dual-write traffic is divided into read and write operation traffic for a phased switchover. Read operation traffic refers to requests for querying or retrieving data in the original business logic. During the dual-write phase, read requests can be switched to the target database first, while monitoring the target database's cache hit rate, response latency, and load to ensure stable read performance. Write operation traffic refers to requests for adding, updating, or deleting data in the original business logic. During the switchover, write operations should be grouped according to transaction identifiers and conflict levels, and gradually switched to the target database to ensure correct data dependencies and avoid operation conflicts. The conflict level is determined based on the distribution of abnormal operations; for example, single-table read operations are low-conflict-level, multi-table join queries are medium-conflict-level, and transactional write operations are high-conflict-level. Switchover priority is determined according to conflict level, with low-conflict-level operations switched first and high-conflict-level operations switched later.

[0178] Monitor the cache status of the target database for read operations. When the cache hit rate for hot data in the target database reaches over 95% and the cache occupancy rate stabilizes at around 60%, the cache status is considered to meet the requirements. At this point, start switching write operation traffic in groups based on transaction identifier and conflict level. For example, first switch write operations with low conflict levels, such as user information updates, with a 30-minute interval between each group of write operation traffic switching, during which system performance is continuously monitored.

[0179] During traffic switching, the incremental verification results in the consistency report are received. When three new inconsistent records are found in the order table, the corresponding transaction identifier (such as TX_ORDER_UPDATE_123) is extracted, and the dual-write traffic of this type of transaction is temporarily adjusted, reducing the traffic of order update operations from the current 40% to 25%, while triggering a targeted data repair process.

[0180] The migration progress is assessed based on the stability of incremental verification results and data synchronization latency. Stability is calculated using the verification pass rate over seven consecutive days. For example, the current seven-day verification pass rates are 98.2%, 98.7%, 99.1%, 99.3%, 99.5%, 99.8%, and 99.9%, showing a stable upward trend. Data synchronization latency has decreased from an initial average of 25 milliseconds to less than 5 milliseconds. When the target database's CPU utilization stabilizes below 65%, memory utilization stabilizes below 70%, query response time stabilizes below 10 milliseconds, and the verification pass rate exceeds 99.5% for 14 consecutive days, the target database is considered to have met performance requirements, and the full migration is completed. At this point, all dual-write traffic has been switched to the target database.

[0181] Through the above implementation methods, a smooth switching of business traffic and dynamic control of migration risks can be achieved during database migration. By analyzing consistency reports and target database performance metrics, dual-write traffic can be adaptively adjusted to ensure that the target database does not experience performance bottlenecks or operational conflicts during synchronization. Grouping and prioritizing read and write operation traffic based on the distribution of abnormal operations and conflict levels can minimize the risk of data inconsistency. Real-time monitoring of incremental verification results and dynamic adjustment of the dual-write strategy can promptly correct abnormal transactions, ensuring data integrity and consistency during the migration process. Overall, this solution achieves high controllability, smoothness, and intelligence in the migration process, and ensures that the target database safely completes the full migration after performance targets are met.

[0182] A second aspect of this invention provides a cross-platform database heterogeneous migration and fault-tolerant control system, the system comprising:

[0183] The first unit is used to build a heterogeneous mapping rule base from the source database to the target database;

[0184] The second unit is used to analyze the access logs of the source database, calculate the data access popularity score using a sliding time window, generate a weighted access popularity list, divide the data into hot datasets and cold datasets according to the access popularity list, and calculate the migration priority for each dataset.

[0185] The third unit is used to establish a task dependency topology graph based on migration priority and heterogeneous mapping rule base, and to arrange the migration task sequence through parallel task scheduling algorithm to form a migration execution scheme that includes data sharding strategy.

[0186] The fourth unit is used to convert the data in the source database according to the rules in the heterogeneous mapping rule base according to the migration execution plan, and then transmit the converted data in fragments to the target database.

[0187] The fifth unit is used to synchronize changes from the source database to the target database in real time using a distributed message queue during transmission. It also performs incremental verification on the synchronized data based on a heterogeneous mapping rule base to ensure data consistency and generates a consistency report. Based on the consistency report, it adopts an adaptive dual-write strategy to smoothly switch business traffic until the target database meets performance requirements and completes the full migration.

[0188] A third aspect of the present invention provides an electronic device, comprising:

[0189] processor;

[0190] Memory used to store processor-executable instructions;

[0191] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0192] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0193] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for cross-platform database heterogeneous migration and fault-tolerant control, characterized in that, The application relates to a method for migrating a source database to a target database. The method comprises the following steps: building a heterogeneous mapping rule library from the source database to the target database; analyzing access logs of the source database, calculating data access heat scores by using a sliding time window, generating a weighted access heat list, dividing data into a hot data set and a cold data set according to the access heat list, and calculating migration priorities for each data set; establishing a task dependency topology graph based on the migration priorities and the heterogeneous mapping rule library, scheduling migration tasks by using a parallel task scheduling algorithm, and forming a migration execution scheme containing a data sharding strategy; according to the migration execution scheme, converting data in the source database into a format according to the rules in the heterogeneous mapping rule library, and transmitting the converted data to the target database; 2. The method of claim 1, wherein, in the transmission process, using a distributed message queue to synchronize source database change information to the target database in real time, ensuring data consistency based on the heterogeneous mapping rule library, performing incremental verification on the synchronized data, and generating a consistency report, using an adaptive double-write strategy to smoothly switch business traffic based on the consistency report, and completing full migration when the target database meets the performance requirements. The method comprises the following steps: obtaining data type information and data operation logs of the source database, extracting data distribution characteristics and field access records; dynamically calculating numerical parameters required by the target storage according to the data distribution characteristics, and generating corresponding numerical conversion rules; constructing a mapping correlation matrix between data structures according to the data distribution characteristics, dividing the conversion mode of the composite structure of the source database by using the mapping correlation matrix, and generating corresponding structure conversion rules; analyzing the dependency relationship between fields by using the field access records, extracting the limitation conditions of field values, and generating corresponding constraint conversion rules; 3. The method of claim 1, wherein, integrating the numerical conversion rules, the structure conversion rules and the constraint conversion rules into the heterogeneous mapping rule library. The method comprises the following steps: obtaining access logs of the source database, extracting access characteristic information and field dependency relationship of data items; determining an initial sliding time window based on the access characteristic information, grouping the access logs according to transaction identifiers in the initial sliding time window, and constructing an access sequence graph reflecting the access order of the data items; analyzing the access time sequence relationship between the data items according to the access sequence graph, identifying the propagation direction of the data items in combination with the field dependency relationship, establishing a hierarchical propagation structure of the data items, calculating the node depth of each data item in the hierarchical propagation structure, and determining the basic access weight of the data item according to the node depth; statistically analyzing the access frequency of the data items in the initial sliding time window, and calculating the access importance of the data items in combination with the basic access weight; dynamically adjusting the size of the sliding time window based on the change trend of the access frequency, determining a differential weight coefficient of the data items in the updated sliding time window according to the access importance of the data items, and calculating and accumulating the access heat scores of the updated data items; The access heat list is generated by ranking the access heat scores, the distribution characteristics of the access heat list are analyzed, the mutation interval of the access heat scores is identified and the demarcation threshold is determined, and the data items are divided into a hot data set and a cold data set according to the demarcation threshold; The migration priorities are calculated based on the access importance, the access heat scores and the access intervals for the hot data set and the cold data set respectively.

4. The method of claim 1, wherein, The migration execution scheme including the data sharding strategy is formed by establishing the task dependency topology graph based on the migration priorities and the heterogeneous mapping rule library, and scheduling the migration task sequence by a parallel task scheduling algorithm, including: The to-be-migrated data items are constructed as task nodes, the priority attributes of the task nodes are determined based on the migration priorities, and the mapping attributes of the task nodes are determined based on the heterogeneous mapping rule library; The dependency relationships between the data items are analyzed, the directed connections are established between the task nodes with dependencies, the weights of the directed connections are calculated based on the priority attributes and the mapping attributes of the task nodes, and the task dependency topology graph is constructed; The task dependency topology graph is topologically sorted to generate a basic migration task sequence; The mapping attributes of the task nodes in the basic migration task sequence are analyzed, the task nodes with the same mapping attributes and no direct dependencies are combined into parallel task groups, the data shard size is determined according to the mapping attributes of the task nodes in the parallel task groups, and an initial data sharding strategy is generated; The system processing capacity is calculated according to the system resource state indicators, the execution parallelism of the parallel task groups is adjusted according to the system processing capacity, and the execution parallelism and the initial data sharding strategy are combined to form the migration execution scheme.

5. The method of claim 1, wherein, According to the migration execution scheme, the data in the source database is format-converted according to the rules in the heterogeneous mapping rule library, and the converted data shards are transmitted to the target database, including: The conversion rules are extracted from the heterogeneous mapping rule library, the conversion rules are parsed into basic conversion units, the basic conversion units are compiled and optimized to generate independent conversion functions, and the complexity weights of each conversion function are calculated; The data sharding strategy in the migration execution scheme is adjusted according to the complexity weights, and the data in the source database is sharded according to the adjusted sharding strategy; A unique identifier is assigned to the data shard, a shard transmission sequence is constructed, the data shard is set with a processing priority based on the complexity weight of the conversion function, and the data shard is read from the source database in order of priority; The integrity feature value of the data shard is calculated, the calculation resources are allocated according to the processing priority of the data shard, and the data shard is converted to the target format by calling the corresponding conversion function; The converted data shard and its integrity feature value are transmitted to the target database, and the integrity of the received data is verified, and the data shard whose verification fails is retransmitted.

6. The method of claim 1, wherein, During the transmission process, the distributed message queue is used to synchronize the source database change information to the target database in real time, and the data that has completed the synchronization is subjected to incremental verification and a consistency report is generated, including: A listener program is deployed in the source database to obtain change information, and the change information includes transaction identifiers, timestamps and change data contents; The change information is sorted according to transaction identification and timestamp, the change information associated with the transaction identification is extracted, the dependency sequence between the change information is identified, the change information with the dependency relationship is combined into an atomic operation unit, the atomic operation unit is executed to generate change data in a standard format through heterogeneous mapping conversion, real-time check values and periodic check values of the change data are calculated, and are associated with the transaction identification and the timestamp and written into a check record table; The atomic operation unit is sharded based on the transaction identification and the dependency sequence, shards with dependencies are allocated to the same message queue, and the priority order and the dependency condition of the shards are recorded in the queue metadata; The multi-threaded consumption of the shard data in the message queue is started, the dependency condition is checked according to the queue metadata, the shards that meet the dependency condition are executed to perform synchronous writing, and the shards that do not meet the dependency condition are set to be delayed and re-queued; The consistency of the shards that complete synchronization is verified based on the real-time check values, and the corresponding records in the target database are executed to perform incremental checking according to the periodic check values, when the incremental checking fails, the associated operations are traced back based on the transaction identification and the timestamp, and the source operations that cause inconsistency are located; The incremental checking results and the abnormal positioning information are summarized to generate a consistency report.

7. The method of claim 1, wherein, Based on the consistency report, an adaptive double-write strategy is used to smoothly switch business traffic until the target database meets the performance requirements to complete full migration, including: Data synchronization delay information, abnormal operation distribution and check failure information in the consistency report are obtained, and performance indicators of the target database are collected; A traffic switching threshold is determined based on the data synchronization delay information and the performance indicators, and a traffic adjustment strategy is developed according to the abnormal operation distribution; The performance indicators of the target database are monitored, and when the traffic switching threshold is met, the double-write traffic is increased, and when it is not met, it is adaptively adjusted according to the traffic adjustment strategy; The double-write traffic is divided into read operation traffic and write operation traffic, the operation conflict level is determined based on the abnormal operation distribution, and the switching priority is divided according to the conflict level; The target database cache state of the read operation traffic is monitored, and when the cache state meets the requirements, the write operation traffic is switched according to the transaction identification and the conflict level; The incremental checking results in the consistency report are received, and when data inconsistency occurs, the corresponding transaction identification is extracted, and the corresponding double-write traffic is adjusted; The migration progress is evaluated according to the stability degree of the incremental checking results and the data synchronization delay information, and when the target database meets the performance requirements, full migration is completed.

8. A system for cross-platform database heterogeneous migration and fault-tolerant control for implementing the method of any of the preceding claims 1-7, characterized in that, It includes: A first unit for constructing a heterogeneous mapping rule library from a source database to a target database; A second unit for analyzing the access log of the source database, calculating the data access heat score using a sliding time window, generating a weighted access heat list, dividing the data into a hot data set and a cold data set according to the access heat list, and calculating the migration priority for each data set; A third unit for establishing a task dependency topology graph based on the migration priority and the heterogeneous mapping rule library, scheduling migration tasks in sequence through a parallel task scheduling algorithm, and forming a migration execution scheme including a data sharding strategy; The fourth unit is configured to perform format conversion on the data in the source database according to the rules in the heterogeneous mapping rule library according to the migration execution scheme, and transmit the converted data to the target database; The fifth unit is configured to, during the transmission, use a distributed message queue to synchronously transmit the source database change information to the target database in real time, perform incremental verification on the synchronized data based on the heterogeneous mapping rule library to ensure data consistency, and generate a consistency report, use an adaptive double-write strategy based on the consistency report to smoothly switch the business traffic until the target database reaches the performance requirement and completes full migration.

9. An electronic device, comprising: The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 7. The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 7. The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 7. ​ 10. A computer-readable storage medium having stored thereon computer program instructions, wherein, ​

Images