A collaborative intelligent verification method and system for full and incremental data migration

Abstract

This invention relates to the field of data migration and data synchronization technology, specifically to a method and system for full and incremental collaborative intelligent verification of data migration. The method establishes a data baseline by performing a full consistency verification after data migration initialization, monitors near real-time data changes by performing incremental consistency verification based on synchronization points and dynamic delay windows, coordinates the two types of verification tasks through a collaborative scheduling mechanism to form a closed-loop process, and finally classifies and automatically compensates and repairs discrepancies in the data. This invention integrates technologies such as hash calculation, point tracking, and dynamic thresholds to achieve integrity, accuracy, and consistency verification throughout the entire data migration lifecycle, reducing resource consumption, decreasing false alarm rates, and improving system reliability and automation levels. It is particularly suitable for data migration scenarios in critical systems such as power grids.

Description

Technical Field

[0001] This invention relates to the field of data migration and data synchronization technology, specifically to a full and incremental collaborative intelligent verification method and system for data migration. Background Technology

[0002] As enterprises accelerate their digital transformation, the demand for cross-system, cross-regional, and cross-platform data migration and real-time synchronization is growing. Especially in critical systems such as power grids, data integrity is directly related to system security and operational stability. During data migration, network anomalies, concurrent write conflicts, and data format conversion errors can lead to data inconsistencies such as missing data, redundancy, inconsistent field values, or logical errors during the migration from the source database to the target database. This can result in serious consequences such as power dispatching errors and abnormal electricity billing for users.

[0003] Traditional verification methods rely heavily on full comparison, which is time-consuming and resource-intensive, making it difficult to meet real-time requirements. Incremental verification has incomplete coverage and is prone to missing edge cases such as "out-of-order transaction commits" and "cross-node synchronization delays" in high-concurrency scenarios. Furthermore, full and incremental verifications operate in isolation and cannot form a closed-loop quality assurance system.

[0004] Therefore, there is an urgent need for an intelligent, high-precision data consistency verification method and system with self-healing capabilities to achieve the evolution from "passive detection" to "proactive prevention + automatic repair", thereby continuously ensuring data integrity throughout the entire migration lifecycle. Summary of the Invention

[0005] This invention aims to provide a comprehensive and efficient intelligent verification method and system for data migration, which integrates full verification, incremental verification and collaborative scheduling mechanism, and combines hash calculation, position tracking, delay window and event secondary verification and other technical means to achieve comprehensive, efficient and low-interference verification of the integrity, accuracy and consistency of data migration process, and supports automatic anomaly location and compensation repair, thereby improving the reliability and automation level of data synchronization system.

[0006] On the one hand, the present invention solves the above-mentioned technical problems through the following technical solution: a full and incremental collaborative intelligent verification method for data migration, the method comprising the following steps:

[0007] S101. After the data migration initialization is completed, a full consistency check is performed to establish a data baseline between the source database and the target database.

[0008] S102. Perform incremental consistency checks based on synchronization points and dynamic delay windows to monitor near real-time data changes;

[0009] S103. The execution order, frequency and resource allocation of full verification and incremental verification tasks are coordinated through a collaborative scheduling mechanism to form a closed-loop verification process.

[0010] S104. Classify the discrepancies identified during verification and perform automatic compensation and repair based on preset rules.

[0011] On the other hand, the present invention also provides a full and incremental collaborative intelligent verification system for data migration, comprising:

[0012] Full verification module: used to perform dynamic sharding primary key comparison, field hash comparison, and aggregate value comparison to generate data baseline;

[0013] Incremental verification module: used to implement site backtracking, real-time secondary verification, delayed window verification, and to mark incremental anomalies;

[0014] The collaborative scheduling module is used to calculate task priorities, adjust verification frequency, trigger scheduling events, and coordinate resource allocation.

[0015] The difference repair module is used to classify differential data, execute compensation strategies, and verify repair results.

[0016] Storage module: Used to store synchronization points, hash baselines, verification logs, and details of difference data;

[0017] Monitoring module: Used to display verification progress, number of anomalies, and system resource usage in real time, and supports alarm notifications.

[0018] The positive and progressive effects of this invention are as follows:

[0019] 1. This invention reduces the resource consumption of full verification by 60% through dynamic sharding, and shortens the time for full comparison of tables with tens of millions of records to hours; the multi-dimensional location tracking and dynamic delay window mechanism of this invention reduces the false alarm rate to below 0.5%, covering edge scenarios such as "delayed commit" and "transaction rollback".

[0020] 2. This invention forms a closed loop between full and incremental verification, reducing the latency of incremental verification of the core table to the second level, taking into account both comprehensiveness and real-time performance. It features a business-aware difference classification and compensation mechanism, automatically repairing more than 80% of inconsistencies, reducing manual intervention, and asynchronous verification, load-aware scheduling, and dynamic resource allocation, which have a minimal impact on the performance of the business system. Attached Figure Description

[0021] Figure 1 The flowchart of a full and incremental collaborative intelligent verification method for data migration provided by the present invention.

[0022] Figure 2 The flowchart for full consistency verification provided by this invention.

[0023] Figure 3 The flowchart for incremental consistency verification provided by this invention.

[0024] Figure 4 The collaborative scheduling logic diagram provided by this invention.

[0025] Figure 5 This is a structural block diagram of the local terminal of an exemplary electronic device of the present invention.

[0026] Figure 6 This is a structural block diagram of the network terminal of an exemplary electronic device of the present invention.

[0027] Figure 7 This is a comparison chart of data consistency trends in this invention.

[0028] Figure 8 This is a comparison chart of the verification time for this invention.

[0029] Figure 9 This is a comparison chart of resource consumption for this invention.

[0030] Figure 10 This is a comparison chart of the anomaly detection performance of the present invention.

[0031] Figure 11 This is a comparison chart of the fault recovery time of the present invention.

[0032] Figure 12 This is a real-time performance monitoring graph for the present invention. Detailed Implementation

[0033] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments.

[0034] The following, in conjunction with the accompanying drawings, provides a more detailed description of the invention, addressing the scenario where, during the migration of a power grid company's core business system from MySQL to a PostgreSQL cloud database, consistency in user electricity usage data (electric_usage, 1 billion records), equipment status data (device_status, 2 million records), and electricity bill settlement data (bill_settlement, 100 million records) needs to be ensured.

[0035] See Figures 1 to 12 As shown, a collaborative intelligent verification method for full and incremental data migration is proposed, which includes the following steps:

[0036] S101. After the data migration initialization is completed, a full consistency check is performed to establish a data baseline between the source database and the target database. The full consistency check includes primary key set comparison under dynamic sharding, field hash comparison, aggregation value, and baseline confirmation. The process includes:

[0037] (1) Full comparison of primary key sets under dynamic sharding: Using the primary key of the table as a unique identifier, construct the primary key sets of the source database and the target database, and find missing and redundant records through set operations.

[0038] Avoid memory overflow through dynamic sharding: For large tables (more than 10 million records), automatic sharding is performed based on the primary key range. The shard size is calculated based on the memory threshold and the number of CPU cores, using the following formula:

[0039] The formula for the fragment size is PS = min(memory constraint (Cm), CPU constraint (Cc));

[0040] The formula for calculating the memory constraint (Cm) is: ;

[0041] Where: M is the total memory of a single node, α is the memory safety threshold (50%~70%), α is the actual available space coefficient (85%-95%, to cope with the reduction of actual available space caused by memory fragmentation), Lavg is the average size of a single record, β is the data inflation coefficient (reserved at 1.1~1.3), and β is the temporary variable coefficient (1.1~1.2, reserving memory for temporary objects during the calculation process). The CPU constraint (Cc) calculation formula is:

[0042] ;

[0043] Where C is the number of effective CPU cores (excluding occupied cores, C = total number of cores × idle rate), and T is the maximum allowed processing time (seconds). Tavg is the CPU utilization coefficient (0.7~0.9, to avoid CPU over-saturation), and Tavg is the average processing time per record (in seconds, which can be obtained through pre-calculated sampling). This represents the parallel overhead factor (1.1~1.2, the additional overhead of multi-threaded switching). This is the burst task factor (1.05~1.1, to handle other tasks preempting the CPU). This ensures that the fragmented data can be loaded into memory and that the parallelism matches the hardware capabilities. `floor` is a function that rounds down to the nearest integer.

[0044] For each table to be validated, execute the following query: SELECT pk FROM table_name WHERE condition; Load the results into memory or a temporary table, and construct Source_PK_Set and Target_PK_Set; Calculate the set difference: Missing records: Missing = Source_PK_Set - Target_PK_Set; Redundant records: Redundant = Target_PK_Set - Source_PK_Set; For the primary key in Missing, retrieve the complete record from the source database and insert it into the target database; For the primary key in Redundant, decide whether to delete or mark it as dirty data based on the business strategy.

[0045] Sharding strategy: For the electricity_usage data table containing 1 billion electricity bill settlement data entries, assuming both the source and target databases are configured with 16-core CPUs and 64GB of memory, the estimated average size of a single record is 1KB, and the actual sampling time for obtaining a single record is 0.000005s. All coefficients are taken as the median value within their respective ranges.

[0046] Memory constraint (Cm) = (64GB × 0.6 × 0.9) / (1KB × 1.2 × 1.1) ≈ 27.45 million rows / fragment;

[0047] CPU constraint (Cc) = (16 cores × 300s × 0.8) / (0.000005s / strip × 1.1 × 1.1) ≈ 634 million stripes / shard;

[0048] The final shard size is 27.45 million records per shard, and it is partitioned twice by user_id%100+date to ensure even sharding.

[0049] Compared to traditional sharding methods, such as fixed shard size (e.g., 1 million records / shard), which do not consider hardware configuration and data characteristics, powerful hardware will result in idle resources, while weak hardware will result in memory overflow / CPU overload.

[0050] like Figure 9 As shown in the resource consumption comparison chart, the dynamic sharding strategy optimized CPU utilization from 85% to 45% using the traditional method and reduced memory usage from 90% to 35%, reserving sufficient resources for other business systems. The bar chart compares the utilization of four resources (CPU, memory, network, and storage) under different verification methods. Collaborative intelligent verification significantly reduced utilization across all resource types, particularly network and storage resources, which saw reductions of 50% and 64%, respectively.

[0051] (2) Full field hash verification method: The key fields of each record are concatenated to generate a hash value. The hash is compared between the source database and the target database to confirm the consistency of the field content.

[0052] Define the list of fields to participate in the hash (excluding non-business fields such as update_time and version);

[0053] For non-core tables, MD5 hashing is used (efficiency first); for core tables (such as order tables), SHA-256 hashing is used (security first). The hash calculation logic is as follows:

[0054] ;

[0055] For large fields such as TEXT / BLOB, a combination of "length + prefix hash" is used for validation;

[0056] col1, col2, col3, ..., colN are the field values ​​that need to be validated. The values ​​of the fields to be validated are concatenated using a delimiter to form a single string. A fixed-length hash value is then generated using a hash algorithm, enabling fast validation of the field values. A string concatenation function with a specified delimiter.

[0057] The formula for large field validation is:

[0058] ;

[0059] The `field` field contains a large amount of data. Due to the length limitation of such a large field, it's impossible to retrieve all the data within the field. Therefore, the complete length of the large field is concatenated with the hash value of the first 1KB of data to form a final verification string. By comparing the hash value of the first 1KB, changes to the content can be detected with a high probability. Here is the variable name used to store the generated validation results, CONCAT is the string concatenation function, LENGTH(field) is the function to calculate the field length, and SUBSTRING is the substring.

[0060] A full field comparison is triggered only when the length or prefix hash is inconsistent, reducing computational overhead.

[0061] Obtain the corresponding hash values ​​of the source and target databases respectively; associate the two sets of hash values ​​by primary key and compare whether they are equal; if they are inconsistent, mark them as "field-level inconsistency" and enter the fine-grained comparison process (field-by-field comparison).

[0062] Multi-dimensional heterogeneous adaptation and accurate comparison

[0063] Primary key set comparison: Set operations were performed on the two shards, and it was found that 1800 records were not synchronized due to the overflow of the bigint type in MySQL in the int4 field in PostgreSQL. The primary key list of the records is missing.

[0064] Field hash comparison:

[0065] Hash calculation for core fields (usage_amount, status):

[0066] MySQL: SHA2(CONCAT(user_id, usage_amount, status, version), 256);

[0067] PostgreSQL: encode(sha256((user_id||usage_amount||status||version)::bytea), 'hex') (replace CONCAT with ||).

[0068] Validation of the large field usage_details:

[0069] CONCAT(LENGTH(usage_details), ':', SHA2(SUBSTRING(usage_details, 1, 1024), 256)), with unified truncation logic at both ends.

[0070] Aggregate value comparison: Calculate SUM(amount) and COUNT(DISTINCTuser_id) for the electricity bill settlement data table bill_settlement. The difference threshold is calculated with a base threshold of 0.01%, and if it is the peak electricity consumption period in summer, it is calculated based on an average fluctuation of 0.01% and a standard deviation of 0.01%.

[0071] Dynamic percentage threshold: 0.01% + (3 * 0.01%) * 1.2 = 0.01% + 0.036% = 0.046%. This means that a significant difference will only be indicated when the difference is greater than 0.046%. During periods of stable electricity consumption, the average fluctuation is <0.001%, and the standard deviation is close to 0. The dynamic percentage threshold is less than the base threshold, so we set it to 0.01%. If only a fixed threshold of 0.01% is used, the results will remain unchanged during periods of stable electricity consumption, but during periods of sudden changes in electricity consumption at the beginning of summer, it will cause unnecessary alarm disturbances. This applies to large tables; similarly, for small tables, changes will result in a larger percentage fluctuation. If the base threshold is used, anomalies will be largely ignored.

[0072] like Figure 7 As shown in the data consistency trend comparison chart, through the multi-dimensional heterogeneous adaptation mechanism, the system maintains a stable data consistency of over 99.8% during 24-hour continuous monitoring, with a fluctuation standard deviation of only 0.2%, which is significantly better than the traditional method of 98.5%±1.0%.

[0073] Hash baseline generation

[0074] Record-level hashing:

[0075] Store pk→Record_Hash by table partitioning (electric_usage is partitioned by date+user_id%100);

[0076] Table-level hash:

[0077] Table_Hash=SHA256(CONCAT(table_name, min_pk, max_pk, MAX(update_time), aggregate checksum));

[0078] like Figure 8 As shown in the comparison chart of verification time, the hash baseline-based verification strategy takes only 8 seconds to process 1 billion records, an 85% improvement over the 120 seconds of traditional full verification, greatly enhancing the efficiency of large-scale data migration. The chart compares the verification time of the three methods under different data scales (from 1,000 to 1 million records). Collaborative intelligent verification shows significant advantages at all scales, especially in the scenario of 1 billion-level electricity bill settlement data, where the time is only 1 / 15 of that of traditional full verification.

[0079] (3) Aggregate value fast verification method (supplementary dynamic threshold algorithm) quickly determines whether there is a major deviation by comparing table-level statistical indicators.

[0080] Commonly used aggregation metrics include: COUNT(*), which can detect overall missing / redundant data; SUM(pk), which detects duplicate or missing primary keys; MAX(update_time), which verifies that the latest data has been synchronized, etc.

[0081] Parallel aggregation queries are executed between the source and target databases. If the difference exceeds a threshold, a sharded hash check is triggered. The threshold is no longer fixed but dynamically adjusted based on historical fluctuations. The calculation formula is: Threshold = max(Base Threshold, μ + 3σ × (1 + D)). Here, μ is the average fluctuation of the indicator over the past 7 days, σ is the standard deviation, the base threshold is the minimum allowable deviation for the business, and D is the data density coefficient, which is 0.5 for small tables and 0.2 for large tables. This avoids frequent misjudgments due to the small amount of data in small tables. For example, if the total amount in a large table has an average fluctuation of 0.005% and a standard deviation of 0.003% over the past 7 days, then the threshold = 0.005% + 3 × 0.003% × 1.2 = 0.0158%, which is higher than the base threshold of 0.01%. The final threshold is set to 0.0158%.

[0082] Baseline confirmation: After the full comparison passes, the current position and hash baseline are recorded as the benchmark for subsequent incremental verification.

[0083] S102. Perform incremental consistency checks based on synchronization points and dynamic delay windows, and monitor near real-time data changes. The process includes:

[0084] (1) Site backtracking: Based on the multi-dimensional site chain of "checkpoint + timestamp", extract all change events of the site to date and verify its final state in the target database.

[0085] Get the last verification point and trace back all DML events after that point; for each event, query the current record status (latest value) in the source database and query the corresponding primary key record status in the target database, and compare whether the field values ​​are consistent; if they are inconsistent, record it as "incremental anomaly" and enter the compensation process.

[0086] (2) Real-time event secondary verification: After the data is successfully written to the target database, a reverse read comparison is immediately initiated to verify whether the writing result is correct.

[0087] After data is successfully written to the target database, an asynchronous verification task is generated (without blocking the main synchronization); records are read from the target database based on the event primary key; a field-level comparison is performed with the original data in the source database; if they do not match, a retry write or alarm notification is immediately triggered; for momentary verification failures (such as when the target database has not completed synchronization), exponential backoff retries are used (retry interval = 1 second × 2). n (n≤5).

[0088] (3) Delay window verification: Set a "stable waiting period" (e.g., 5-15 minutes) to avoid misjudgment of temporary inconsistencies caused by long transactions and network delays.

[0089] Set the dynamic delay window delay_window(window = max(base window, dynamically adjusted window)), where the formula for calculating the dynamically adjusted window is:

[0090] ;

[0091] ;

[0092] ;

[0093] The average synchronization delay is 1 hour. The standard deviation is used to record `sync_finish_time` after all incremental events have been written. The scheduler periodically scans for events whose `sync_finish_time + delay_window` has expired and performs a consistency comparison on the records corresponding to these events. If inconsistencies are still found, it is determined as a "real anomaly". The dynamically adjusted window value is calculated, where k is the time period number, t is the time interval for a single sampling, kt is the current time node, and D...(k-n+1)t,kt Let t be the delay from time (k-n+1)t to time kt.

[0094] (4) Anomaly marking: For records that are confirmed to be inconsistent, mark the difference type (not synchronized / synchronization error / logical conflict) and record detailed information (primary key, field difference, location information).

[0095] 1. Multi-dimensional site tracking

[0096] Maintain the mapping relationship between MySQL binlog position, PostgreSQL LSN and business timestamp (update_time) to ensure cross-database position alignment.

[0097] 2. Real-time and Delay Verification

[0098] Real-time secondary verification: Real-time verification is enabled for the bill_settlement table of electricity bill settlement data. After each deduction record is written to PostgreSQL, the hashes of the amount and status fields in the source database and the target database are immediately compared.

[0099] Dynamic delay window calculation: Synchronization delay data is collected every 10 seconds within 1 hour. For each calculation, the 360 ​​most recently collected delay data points are selected, and the arithmetic mean and standard deviation are calculated based on these 360 ​​data points. Under stable conditions, the delay time remains stable at 210 seconds, with a maximum single fluctuation of ±20 seconds. Based on the average synchronization latency and standard deviation within one hour, the dynamic window (210 + 3 × 20 = 270 seconds) is less than the set 300 seconds. Therefore, a secondary check is performed on records that have been synchronized for more than 5 minutes. In the event of a sudden anomaly in heterogeneous synchronization, such as a field mapping error, the 200th-300th data point (2000-3000 seconds, 100 data points) experiences a conversion failure and retry, causing the latency to spike to 500 seconds. The remaining 300 data points still have a latency of 180-240 seconds (anomaly rate ≈ 27.8%). The overall average is approximately 275.91 seconds, the overall standard deviation is approximately 122.79 seconds, and the dynamic window (≈ 644.28) is greater than the base window. Therefore, a secondary check is performed on records that have been synchronized for more than 11 minutes. This approach ensures efficiency when the system is running smoothly (low latency, low load, no faults) and allows sufficient time for the system to process backlogged tasks when the system experiences sudden congestion (latency spikes, queue accumulation), thus preventing data inconsistency.

[0100] like Figure 12As shown in the real-time performance monitoring graph, this dual-axis graph displays the processing latency (left axis) and error rate (right axis) monitoring over 24 hours. Collaborative intelligent verification reduced latency from an average of 5 seconds to 2 seconds (a 60% reduction) and the error rate from 5% to 1% (an 80% reduction). The dynamic latency window mechanism further reduced processing latency from an average of 5 seconds to 2 seconds (a 60% reduction) and the error rate from 5% to 1% (an 80% reduction) using traditional methods, maintaining a 210-second window during stable periods and automatically expanding to a 644-second window during abnormal periods.

[0101] Anomaly detected: During delayed verification, three meter reading data entries were found to have inconsistent usage_time fields due to time zone differences, and were marked as "synchronization error difference".

[0102] like Figure 10 The anomaly detection performance comparison chart shows that incremental verification achieves an anomaly detection accuracy of 96%, more than 20% higher than traditional methods. The chart illustrates the detection accuracy of four anomaly types (data loss, synchronization errors, version conflicts, and logical errors) under different verification methods. Collaborative intelligent verification achieves over 90% accuracy across all anomaly types, a 25% improvement over traditional methods.

[0103] 3. Heterogeneous model adaptation processing

[0104] String concatenation: uniformly convert to user_id|usage_amount format. MySQL uses CONCAT_WS, and PostgreSQL uses || concatenation followed by hashing.

[0105] Time type: Convert MySQL datetime and PostgreSQL timestampwithtimezone to UTC timestamp before calculating hash to avoid time zone discrepancies.

[0106] S103. The execution order, frequency, and resource allocation of full verification and incremental verification tasks are coordinated through a collaborative scheduling mechanism to form a closed-loop verification process, specifically including:

[0107] (1) Initialization phase: After the data migration is completed, a full verification is performed immediately to establish a baseline;

[0108] (2) Daily operation: Start the incremental verification service, including real-time secondary verification (enabled for core tables), delayed window verification (enabled by default for all tables) and position backtracking verification once per hour;

[0109] (3) Scheduled tasks: Full verification is dynamically triggered based on table priority - full aggregate verification (quick screening) is performed every morning at midnight, and full hash verification (deep comparison) is performed every Sunday;

[0110] (4) Dynamic adjustment: Based on priority scoring (priority = business importance × 5 + data volume × 3 + change frequency × 2), tasks are sorted and core tables are executed first; the frequency is adjusted according to the historical anomaly rate (encrypted verification when the anomaly rate is > 5%), and additional verification is triggered after the synchronization delay exceeds the standard and the node is restored.

[0111] 1. Priority Scheduling

[0112] Task priority score: bill_settlement (electricity bill settlement) = 5 (business importance) + 3 (data volume) + 2 (change frequency) = 10 points, device_status = 9 points, core tables are prioritized for scheduling.

[0113] Resource control: During peak hours (8:00-22:00), the parallelism of incremental verification is ≤2 threads, and during off-peak hours it is increased to 8 threads.

[0114] 2. Triggered scheduling

[0115] Scheduled tasks: Perform full aggregate verification at 0:00 every day and full hash verification at 0:00 every Sunday.

[0116] Anomaly-driven: When the device status synchronization delay is greater than 5 minutes (more than twice the threshold), the incremental verification of the device_status table is automatically triggered.

[0117] S104. Classify the discrepancies identified during verification and perform automatic compensation and repair based on preset rules. The process includes:

[0118] Difference Classification

[0119] The rule engine determines the type of difference: unsynchronized difference (source has it but target does not and the position is not covered), synchronization error difference (hash mismatch and the position has been covered), and logical conflict difference (both versions are higher than the last verification).

[0120] Unsynchronized discrepancies: 1800 electricity usage records were not synchronized due to type overflow;

[0121] Synchronization error discrepancies: 3 records of inconsistent usage_time due to time zone conversion;

[0122] Logical conflict: Version conflict caused by concurrent updates to the user account table.

[0123] Automatic compensation

[0124] Match repair rules from the compensation strategy library, such as rewriting unsynchronized records, resynchronizing erroneous synchronization records, and selecting the best conflict records by timestamp;

[0125] Unsynchronized differences: The type conversion interface (MySQL bigint → PostgreSQL int8) was called to batch write 1800 records;

[0126] Synchronization error difference: After performing a unified time zone conversion (to UTC time), the target database records are rewritten;

[0127] Logical conflict: Retain the latest record based on update_time.

[0128] like Figure 11 The fault recovery time comparison chart shows that the automatic compensation mechanism achieves second-level recovery: 15-second recovery for node failure (traditional 180 seconds, a 92% improvement), 10-second recovery for network interruption (traditional 120 seconds, a 92% improvement), 30-second recovery for data conflict (traditional 240 seconds, an 88% improvement), and 45-second recovery for batch errors (traditional 300 seconds, an 85% improvement). This chart compares recovery times under four fault scenarios (node ​​failure, network interruption, data conflict, and batch errors). Collaborative intelligent verification reduces recovery time from minutes to seconds, with a maximum improvement of 85%.

[0129] 3. Repair confirmed

[0130] Perform batch rewriting (1000 records per batch, executed asynchronously), call the synchronous API to resynchronize (retry 3 times), or retain the latest version based on the timestamp; after compensation is completed, perform secondary verification to confirm the difference repair; update the site records and enter the next round of verification loop.

[0131] Furthermore, this invention also provides a collaborative intelligent verification system for full and incremental data migration, comprising:

[0132] Full verification module: used to perform dynamic sharding primary key comparison, field hash comparison, and aggregate value comparison to generate data baseline;

[0133] Incremental verification module: used to implement site backtracking, real-time secondary verification, delayed window verification, and to mark incremental anomalies;

[0134] The full verification module provides a foundation of trust for the incremental verification module: the data baseline (hash value, aggregate value) established by the full verification is not statically archived; when the incremental verification module performs position backtracking or real-time verification, it will first query the baseline data in the storage module; for fields that have not changed, the baseline hash can be directly used for comparison, avoiding repeated calculations and greatly improving the efficiency of incremental verification; this reflects the collaboration of full verification laying the foundation for incremental verification; and the incremental verification module provides optimization guidelines for the full verification module: the incremental verification module continuously monitors data change hotspots (such as which tables and which fields are updated most frequently); this information is fed back to the collaborative scheduling module, which in turn guides the dynamic sharding strategy of the full verification module; for tables that change frequently, the shard size (PS) can be automatically reduced in the next full verification, speeding up the verification speed of this part and ensuring that the baseline can be updated faster, forming a collaboration of "incremental feedback guiding full optimization";

[0135] The collaborative scheduling module is used to calculate task priorities, adjust verification frequency, trigger scheduling events, and coordinate resource allocation.

[0136] The difference repair module is used to classify differential data, execute compensation strategies, and verify repair results.

[0137] After the difference repair module performs compensation, it does not simply end; it automatically generates a micro-verification task for the batch of difference data and submits it to the incremental verification module for immediate verification; the verification result is simultaneously fed back to the collaborative scheduling module (used to evaluate the effectiveness of the repair strategy) and the monitoring module (used to update the repair success rate dashboard); this constitutes a complete closed-loop collaboration of "discovery-repair-verification";

[0138] Storage module: Used to store synchronization points, hash baselines, verification logs, and details of difference data;

[0139] The monitoring module is used to display the verification progress, number of anomalies, and system resource usage in real time, and supports alarm notifications. The monitoring module displays in real time when the full verification is scanning a large table shard, causing the CPU utilization to reach 80%. The collaborative scheduling module can issue instructions in real time to temporarily reduce the number of parallel threads of the incremental verification module, or suspend the real-time secondary verification of non-core tables to ensure that the performance of the core business synchronization link is not affected.

[0140] The collaborative scheduling module automatically triggers incremental verification after synchronization delay exceeds twice the threshold, node failure recovery, or batch import. During off-peak periods, it increases the full verification resource quota to reduce the impact on business system performance. The collaborative scheduling module can learn from historical data and establish a dynamic triggering model. For example, if it is found that during the system batch backup period, the synchronization delay increases periodically but does not necessarily lead to differences, the trigger threshold can be temporarily increased during this window to avoid invalid verification. During the settlement period after the business peak, the trigger threshold is automatically reduced to improve verification sensitivity.

[0141] Taking the scenario of "resolving the issue of 1800 records not being synchronized due to data overflow in electricity billing" in the power grid migration scenario as an example, the collaborative work of the above modules is as follows:

[0142] Problem identified:

[0143] The full verification module, through dynamic sharding primary key comparison, initially detected 1,800 missing records and stored the difference details in the storage module;

[0144] The collaborative scheduling module determines a batch difference as a high-priority difference based on the rule ("unsynchronized difference" + "data volume > 1000") and immediately triggers the difference repair module.

[0145] Meanwhile, to prevent new business changes from occurring during the repair process due to these missing records, the collaborative scheduling module notifies the incremental verification module to add the primary keys of these 1,800 records to the "key attention list". In subsequent incremental verifications, once a change event of these records in the source database is detected, a strong consistency real-time secondary verification will be performed.

[0146] Analyze and fix:

[0147] The difference repair module reads the difference details from the storage module and matches the compensation strategy for "unsynchronized difference - type mismatch" according to the preset rules (calling the type conversion interface to batch rewrite).

[0148] Before performing the write operation, the difference repair module queries the current resource status of the monitoring module through the collaborative scheduling module and selects a time when the target library's CPU and I / O are idle to initiate the batch write operation.

[0149] During the rewriting process, the status of each step (success, failure, retry) is written back to the repair log of the storage module in real time;

[0150] Verification and closed-loop:

[0151] After the batch rewriting is completed, the difference repair module automatically submits an "instant verification task" for these 1,800 primary keys to the incremental verification module;

[0152] The incremental verification module performs this task, retrieving data from the source and target databases respectively for comparison to confirm consistency.

[0153] The verification results (1800 successful records) are fed back to the difference repair module, marking the repair task as complete, and the difference record status in the storage module is updated synchronously.

[0154] On the alarm dashboard of the monitoring module, the related anomaly count is reset to zero, and a system log message "Batch differences have been automatically repaired and verified" is generated, completing the entire collaborative processing loop.

[0155] The present invention also discloses a machine-executable program that can be automatically executed by a machine to realize the full and incremental collaborative intelligent verification method for data migration as described above.

[0156] The machine (electronic device) mentioned above is, for example, a microcontroller, a single-board computer, a desktop computer, a laptop computer, a server, a programmable controller, or a field-programmable gate array.

[0157] Figure 5 This is a structural block diagram of the local terminal of an exemplary electronic device (machine) of the present invention; as shown... Figure 5 As shown, the electronic device of the present invention includes a processor 31, a memory 32 and a storage space 33 for storing a machine-executable program 34, the machine-executable program 34 being used to execute the above-described control logic.

[0158] Figure 6 This is a structural block diagram of the network end of an exemplary electronic device of the present invention; as shown below. Figure 6 As shown, the present invention also provides an electronic device (machine), which may include at least one processor 410, at least one memory 430 communicatively connected to the processor, and a communication bus 440 connecting different system components (including memory 430 and processor 410); wherein, memory 430 stores a machine executable program that can be executed by the processor, and processor 410 can execute the above-mentioned control logic by calling the machine executable program.

[0159] Communication bus 440 represents one or more of several bus architectures, including a memory bus or memory controller, peripheral bus, graphics acceleration port, processor, or local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnection (PCI) bus.

[0160] Electronic devices typically include a variety of computer system readable media, which can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0161] Memory 430 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 430 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the control logic described above.

[0162] A program / utility having a set (at least one) of program modules can be stored in memory 430. Such program modules include, but are not limited to, an operating system, one or more applications, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0163] Machine-executable programs for performing this invention can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, C++, and Python, and may also include specialized engineering languages ​​such as R. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0164] The present invention also discloses a storage medium on which a machine-executable program as described above is stored.

[0165] The aforementioned storage medium may be any combination of one or more computer-readable media. Computer-readable media may be, for example, computer-readable signal media or computer-readable storage media. Computer-readable storage media include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium may be, for example, any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0166] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0167] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0168] This invention is not limited to the embodiments described above. Any changes in shape or structure shall fall within the protection scope of this invention. The protection scope of this invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of this invention, but all such changes and modifications shall fall within the protection scope of this invention.

Claims

1. A collaborative intelligent verification method for full and incremental data migration, characterized in that, The method includes the following steps: S101. After the data migration initialization is completed, a full consistency check is performed to establish a data baseline between the source database and the target database. S102. Perform incremental consistency checks based on synchronization points and dynamic delay windows to control long transactions and network latency, perform consistency comparisons on records corresponding to events, and monitor near real-time data changes. In the aforementioned delay window verification, the dynamic delay window is set to max(base window, dynamically adjusted window), and the formula for calculating the dynamically adjusted window is: ； ； ； in, The average synchronization delay is 1 hour. Let n be the standard deviation, and n be the number of samples taken within one hour. This represents the delay value at the i-th measurement point. The dynamically adjusted window value is calculated. S103. The execution order, frequency and resource allocation of full verification and incremental verification tasks are coordinated through a collaborative scheduling mechanism to form a closed-loop verification process. S104. Classify the discrepancies identified during verification and perform automatic compensation and repair based on preset rules.

2. The intelligent verification method for full and incremental data migration as described in claim 1, characterized in that: The full consistency check in S101 includes primary key set comparison under dynamic sharding, field hash comparison, aggregate value, and baseline confirmation. In the primary key set comparison under dynamic sharding... The formula for fragment size is PS = min(memory constraint (Cm), CPU constraint (Cc)); The formula for calculating the memory constraint (Cm) is: ; The formula for calculating CPU constraint (Cc) is: ; Where M is the total memory of a single node, α is the memory safety threshold (actual usable space coefficient is 85%-95%), Lavg is the average size of a single record, and β is the data inflation coefficient. For temporary variable coefficients: reserve memory for temporary objects during the calculation process; C is the effective number of CPU cores: excluding occupied cores, C = total number of cores × idle rate; T is the maximum allowed processing time. Tavg is the CPU utilization coefficient; Tavg is the average processing time per record. Parallelism overhead: the additional overhead of multi-threaded switching. `floor` is the burst task coefficient, used to handle other tasks that may preempt the CPU; `floor` is a function that rounds down to the nearest integer.

3. The full and incremental collaborative intelligent verification method for data migration as described in claim 2, characterized in that: The field hash comparison uses an adaptive hash algorithm: MD5 for non-core tables and SHA-256 for core tables. The logical formula for hash calculation is: ； col1, col2, col3, ..., colN are the field values ​​that need to be validated. The values ​​of the fields that need to be validated are connected together using a delimiter to form a single string, and a fixed-length hash value is generated by a hash algorithm, so as to perform fast validation of the field values. A string concatenation function with a specified delimiter; The formula for large field validation is: ； The `field` field contains a large amount of data. Due to the length limitation of such a large field, it's impossible to retrieve all the data within the field. Therefore, the complete length of the large field is concatenated with the hash value of the first 1KB of data to form a final verification string. By comparing the hash value of the first 1KB, changes to the content can be detected with a high probability. Here is the variable name used to store the generated validation results, CONCAT is the string concatenation function, LENGTH(field) is the function to calculate the field length, and SUBSTRING is the substring.

4. The intelligent verification method for full and incremental data migration as described in claim 2, characterized in that: The comparison difference threshold of the aggregated value is max(base threshold, μ + 3σ × (1 + D)), where μ is the historical average fluctuation value, σ is the standard deviation, and D is the data density coefficient.

5. The intelligent verification method for full and incremental data migration as described in claim 1, characterized in that: The incremental consistency check in S102 includes site backtracking, real-time event secondary check, delayed window check, and anomaly marking. The real-time event secondary check uses an exponential backoff retry algorithm, with a retry interval of initial_delay × 2. n n is the number of retries, and initial_delay = 1 second.

6. The intelligent verification method for full and incremental data migration as described in claim 1, characterized in that: The cooperative scheduling mechanism in S103 includes: Task priority scoring: Priority score is calculated based on business importance, data volume and change frequency. The formula is: Priority score = (Business importance weight × 5) + (Data volume weight × 3) + (Change frequency weight × 2). Dynamic frequency adjustment: The full verification frequency is adjusted based on the historical anomaly rate. It is executed daily when the anomaly rate is >5% and weekly when the anomaly rate is <1%. The incremental verification frequency is adjusted based on the change frequency. It is executed every 5 minutes for high-frequency tables and every hour for low-frequency tables. Triggered scheduling: Incremental verification is automatically triggered when the synchronization delay is greater than twice the threshold, after a node failure is recovered, or after batch import.

7. A system for a collaborative intelligent verification method for full and incremental data migration as described in any one of claims 1-6, characterized in that: include: Full verification module: used to perform dynamic sharding primary key comparison, field hash comparison, and aggregate value comparison to generate data baseline; Incremental verification module: used to implement site backtracking, real-time secondary verification, delayed window verification, and to mark incremental anomalies; The collaborative scheduling module is used to calculate task priorities, adjust verification frequency, trigger scheduling events, and coordinate resource allocation. The difference repair module is used to classify differential data, execute compensation strategies, and verify repair results. Storage module: Used to store synchronization points, hash baselines, verification logs, and details of difference data; Monitoring module: Used to display verification progress, number of anomalies, and system resource usage in real time, and supports alarm notifications.

8. The system as described in claim 7, characterized in that: The collaborative scheduling module automatically triggers incremental verification after synchronization delay > 2 times the threshold, node failure recovery, or batch import. During off-peak hours, it increases the full verification resource quota to reduce the impact on business system performance.

Images