Database fast migration method and system based on edge computing

Abstract

The application discloses a kind of based on edge computing's database fast migration method and system, method core step includes: topology perception: detection source / target node resource, network path and database information;Intelligent fragmentation and routing planning: based on reinforcement learning dynamic optimization data piece transmission path;Low overhead CDC and multistage cache: through transaction log analysis capture change, combined with memory / persistent queue management incremental data;Parallel transmission and three layers compression: establish multi-channel transmission, idempotent write and check;Near-zero downtime switching: suspend source base write-in→flat last increment→rapid check→switch flow;Resource recovery: clean migration state.Through dynamic fragmentation, multi-channel transmission and adaptive compression, migration efficiency is improved by more than 30%;CDC and full-parallel processing, downtime is compressed to seconds;Support network interruption recovery and heterogeneous database environment, guarantee strong consistency;Resource consumption is reduced by 50%, adapt to edge device limit.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for rapid database migration based on edge computing. The method and system are suitable for completing the migration of large-scale databases from a source end to a target end in a distributed edge computing environment with high efficiency, low latency, and high reliability, and are particularly adaptable to environments with limited edge node resources and dynamically changing network conditions. Background Art

[0002] With the explosive growth of applications such as the Internet of Things, Industrial Internet, and Smart Cities, data processing needs are shifting from cloud computing centers to the edge of the network. Edge computing, by deploying computing and storage resources locally, provides low-latency, high-bandwidth data processing capabilities and has become a core support for real-time applications. In this context, databases need to adapt to edge architectures. Typical scenarios include:

[0003] Cloud databases are deployed to edge nodes (reducing latency and alleviating pressure on the core network); migration / synchronization between edge nodes (due to device movement, load balancing, or failover); and multi-level edge-cloud data collaboration (on-demand hierarchical migration and synchronization).

[0004] Unique Challenges of Database Migration at the Edge

[0005] 1. Complex and dynamic network

[0006] High latency and fluctuation: Cross-region / carrier node latency is significantly higher than in data centers, especially in wireless networks. Limited and unstable bandwidth: Edge networks (such as the last mile) have low bandwidth and are susceptible to congestion. Asymmetric bandwidth: Uplink bandwidth is significantly smaller than downlink bandwidth, creating a transmission bottleneck. Intermittent connectivity: Mobile nodes may frequently disconnect from the network.

[0007] 2. Strictly limited edge node resources

[0008] Weak computing power: Edge devices (gateways / embedded devices) have CPUs that cannot handle the high overhead of traditional migration tools (such as data serialization / compression). Small memory capacity: Difficulty caching large data or maintaining complex migration states. Low storage performance: Limited local storage IOPS / throughput affects migration efficiency.

[0009] 3. Highly heterogeneous environment

[0010] The hardware architecture (x86 vs ARM), software version, storage engine, and operating system of the source / target database may differ significantly.

[0011] 4. High business continuity requirements

[0012] Scenarios such as industrial control and the Internet of Vehicles require near-zero downtime migration, with pause times compressed to seconds. Strong data consistency must be ensured even in the event of network outages or node failures.

[0013] 5. Huge data scale

[0014] The Internet of Things generates TB-PB-level time series / log data, and the migration window is limited.

[0015] Existing technical solutions and defects

[0016] Logical export / import based on database-native tools: Disadvantages: Inefficiency, especially for large tables; High CPU overhead; Often requires read-only or downtime; Single-threaded transfer; Difficulty in fault tolerance. Database physical backup and recovery: Disadvantages: Requires downtime to obtain consistent backups; Large backup files, time-consuming to restore; Dependency on specific file systems. Logical / physical replication: Disadvantages: Initial synchronization is a bottleneck, resulting in long latency after network outages; Complex configuration; Performance overhead associated with continuous replication. File / block-level synchronization tools: Disadvantages: Inability to guarantee internal database consistency; Low transfer efficiency; Resource-intensive initial synchronization.

[0017] Existing technologies are unable to effectively solve the core pain point of database migration in edge computing environments: how to achieve fast, near-zero downtime, highly reliable, and low-resource migration of massive data between edge nodes with limited resources and poor network conditions. There is an urgent need for an innovative database migration method and system specifically optimized for edge computing environments. Summary of the Invention

[0018] The present invention addresses the problems of high latency, low bandwidth, limited resources, heterogeneous environments, and high business continuity requirements faced by existing technologies when migrating databases in edge computing environments, and provides a method and system for rapid database migration based on edge computing. The method and system aim to: significantly reduce the total time for database migration, especially the synchronization time for initial large data volumes. Minimize the impact of the migration process on the performance of the source and target databases, and achieve near-zero downtime migration. Efficiently utilize the limited and unstable network bandwidth between edge nodes. Adapt to the limited computing, memory, and storage resources of edge nodes. Ensure strong consistency and high reliability of data during the migration process, and tolerate network interruptions and temporary node failures. Support heterogeneous database source and target environments.

[0019] To achieve the above objectives, the present invention provides the following technical solution: a database rapid migration method based on edge computing, comprising the following steps:

[0020] S1: Receives a migration task request, deploys a source-side edge migration agent (EMA) on the source edge node, deploys a target-side EMA on the target edge node, performs edge topology awareness, collects information, and reports it to the migration coordination center (MCC);

[0021] S2: The MCC performs intelligent data sharding to generate multiple data slices. It then runs a dynamic path optimization algorithm to plan the optimal transmission path for each data slice, and formulates a migration plan for distribution.

[0022] S3: Based on the migration plan, the source-side EMA initiates low-overhead change data capture (CDC) and manages changed data using a multi-level cache, while also performing progressive reads by data slice.

[0023] S4: The source-side EMA establishes parallel multi-transmission channels, applies a three-layer adaptive compression strategy to the read data blocks or CDC change batches, and reliably transmits them through the planned optimal transmission path.

[0024] S5: The target EMA receives and decompresses the data, performs batch writes on the full data, sorts the incremental changes by transaction, and applies them idempotently to the target database.

[0025] S6: Perform data verification and execute near-zero downtime switchover after the data is consistent;

[0026] S7: After the migration is completed, resources are recycled and cleaned up.

[0027] Preferably, the method further comprises the following steps:

[0028] The edge topology perception includes: detecting the CPU, memory, disk performance and capacity information of the source edge node and the target edge node; detecting the direct or indirect network path between the source edge node and the target edge node; and obtaining the source database and target database information.

[0029] Preferably, the method further comprises the following steps:

[0030] The dynamic path optimization algorithm is a model or heuristic algorithm based on reinforcement learning (Q-Learning). It continuously evaluates path status and node load and dynamically adjusts the routing of data slices. The adjustment strategies include: switching to a backup path when the primary path is congested; adjusting the sending rate based on the target node storage IO load; and limiting the number of parallel transmission channels.

[0031] Preferably, the low-overhead CDC includes: prioritizing parsing database transaction logs to capture changes; first storing change events in a memory queue sorted by transaction ID and timestamp; when the memory queue reaches a threshold, asynchronously persisting older untransmitted events to local storage; and maintaining a global change bitmap (GCB) or a Cuckoo Filter variant to record the range of changed data.

[0032] Preferably, the reliable transmission includes: using the QUIC protocol or TCP with BBR to optimize congestion control; implementing block confirmation and selective retransmission; persistently recording the block transmission status of each data slice / CDC batch to achieve breakpoint resumption; setting a higher transmission priority for CDC change data than for full data; and performing bandwidth shaping according to the MCC policy.

[0033] Preferably, the three-layer adaptive compression strategy is specifically:

[0034] Algorithm selection layer: The Zstandard algorithm is used for text / structured data when the CPU load is low, and the LZ4 algorithm is used when the CPU load is high. Binary LOBs are selectively compressed after evaluation.

[0035] Compression level tier: Use high compression level when the network is idle or there is ample migration time, and use low compression level when the network is congested or time is tight;

[0036] Block size layer: Reduce the pre-compression data block size when RTT / packet loss is high, and increase the block size when RTT / packet loss is low.

[0037] Preferably, the idempotent application of incremental change data includes: using an INSERT ... ON DUPLICATE KEYUPDATE statement or precise positioning based on a complete row image; ensuring atomic application of changes within transaction boundaries; and maintaining an application bitmap to record transaction log locations that have been successfully applied.

[0038] Preferably, the data verification includes: shard-level compact checksum (XXH3, CityHash) comparison after each shard is completed in the full migration; row-level sampling verification or change summary comparison based on the global change bitmap (GCB) / CuckooFilter in the incremental migration phase; and final consistency verification before switching.

[0039] Preferably, the near-zero downtime switching includes: pausing source database write operations; transmitting and applying the last incremental changes generated during the pause; performing a fast final check; switching application traffic to the target database; and providing a fast rollback mechanism designed based on the application layer.

[0040] The present invention provides a database rapid migration system based on edge computing, which is used to implement any of the above methods, including:

[0041] Migration Coordination Center (MCC): Deployed in the edge cloud or central cloud; includes a task management module, topology discovery and monitoring module, intelligent planning engine, global scheduler, monitoring and alarm center, and API gateway;

[0042] It also includes Edge Migration Agent (EMA): a lightweight software agent deployed on the source edge node and the target edge node; including local resource and network probes, database connectors, data sharding processor (source side), change data capture (CDC) engine (source side), transmission engine (source side), reliable transmission controller, traffic shaper, receiving and decompression engine (target side), data writing engine (target side), local state manager, heartbeat and reporting module.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. Intelligent sharding and parallel multi-channel transmission fully utilize available bandwidth and node resources. Three-layer adaptive compression greatly reduces network transmission volume and achieves the best balance between compression efficiency and speed. Dynamic path optimization avoids congested links and selects the optimal path. Progressive reading and batch writing reduce memory pressure and improve IO efficiency.

[0045] 2. Full and incremental migrations are performed in parallel, with CDC initiated at the start of the migration, significantly shortening the final incremental catch-up time. Low-overhead CDC, based on transaction log analysis, minimizes the impact on source database performance. Resource-aware scheduling prevents migration tasks from exhausting edge node resources, ensuring the normal operation of business applications. Bandwidth shaping prevents migration traffic from crowding out business bandwidth. Optimized switching processes reduce application downtime to seconds.

[0046] 3. High reliability and consistency. Multi-level change caching ensures that CDC data is not lost in the event of network interruption or node failure. Reliable transmission protocols ensure complete data delivery. Idempotent change applications and transaction atomicity ensure that target database data is ultimately strongly consistent even in the event of retransmission or repeated reception. Multi-level data verification (shard verification, incremental summary, and final verification) ensures data accuracy.

[0047] 4. Efficient resource utilization and adaptability: lightweight EMA design with low CPU / memory usage, adapting to resource-constrained edge devices; dynamically adjusting shard size, compression strategy level, and transmission block size to adapt to changing network conditions and node loads; and utilizing edge node local storage as a cache / queue to alleviate memory pressure.

[0048] 5. Supporting heterogeneous environments, EMA supports multiple mainstream databases; the intelligent planning engine considers differences in source / target hardware and database types; the transport layer is abstracted to shield underlying network differences.

[0049] 6. Simplify operations and maintenance. MCC provides a unified control plane and visual monitoring to simplify large-scale edge environment migration management. A high degree of automation reduces manual intervention. A comprehensive alarm and logging mechanism is also provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 : Overall architecture diagram of the database rapid migration system based on edge computing.

[0051] Figure 2 : Flowchart of the main steps of the migration method.

[0052] Figure 3 : Schematic diagram of intelligent data sharding and dynamic path optimization.

[0053] Figure 4 :Source-side multi-level change cache (CDC) structure diagram.

[0054] Figure 5 : Schematic diagram of three-layer adaptive compression strategy.

[0055] Figure 6 : Schematic diagram of reliable transmission mechanism. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] Example 1, please refer to Figures 1-6 The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. A database rapid migration method based on edge computing includes the following steps:

[0058] S1: Migration task initialization and topology awareness

[0059] This phase begins with receiving a migration task request. The request must clearly specify the source database (located in the source edge node, cloud, or data center), the target database (located in the target edge node), and the migration scope (for example, full, partial tables, or databases).

[0060] The system then deploys a lightweight Edge Migration Agent (EMA) on the source and target edge nodes (if not already deployed). EMA is the core component for performing subsequent migration operations.

[0061] After deployment, EMA will perform edge topology awareness, automatically detect and collect key environmental information, including: source / destination node resources: CPU model and number of cores, available memory, disk type / IOPS / throughput / remaining space, and network interface type.

[0062] Network Path Information: Direct or indirect (possibly via other edge nodes or gateways) network paths from source nodes to target nodes; measure real-time or historical round-trip delay (RTT), available bandwidth (uplink / downlink), packet loss rate, and jitter for each path; identify the optimal path and backup path.

[0063] Database Information: Database type (such as MySQL, PostgreSQL, Oracle, etc.), version, key configuration parameters, character set, table structure (DDL), data distribution statistics (such as table size, estimated number of rows).

[0064] Finally, EMA will report all the information collected to the Migration Coordination Center (MCC). MCC is usually deployed in regional edge clouds or central clouds.

[0065] S2: Intelligent Data Sharding and Routing Planning

[0066] Based on the information collected in S1, the MCC (Migration Control Center) performs intelligent data migration planning, mainly including two core tasks:

[0067] Data Sharding: intelligently divide the data set (such as tables) to be migrated into multiple smaller, logically independent data pieces. MCC supports flexible sharding strategies: primary key / unique key range sharding: especially suitable for large tables, divided according to the range of primary keys or unique keys; Hash sharding: based on the hash value of the specified column, ensuring uniform distribution of data among each shard; list sharding: divided according to the value list of a specific column; mixed strategy: a combination of multiple strategies, aiming to balance data distribution uniformity, shard size balance, and migration efficiency. It is worth noting that the size of the shard is not fixed, but dynamically adjusted according to real-time network bandwidth and node resource conditions.

[0068] Routing Planning: Determine the optimal transmission path for each data piece. When planning, fully consider the following key factors: real-time network quality of the path (including delay, available bandwidth, packet loss rate); real-time load of EMA (Migration Agent) on the source node and target node (covering CPU, memory, disk IO, network IO); historical stability of the path (such as past interruption frequency); storage load of the target node.

[0069] Dynamic Path Optimization: The core of MCC is its dynamic path optimization algorithm (such as a model based on reinforcement learning Q-Learning or an efficient heuristic algorithm). This algorithm continuously evaluates the network path state and the load of each node, and dynamically adjusts the routing of data pieces accordingly. For example: when the main transmission path is congested, the system will automatically switch newly generated data pieces to the backup path; or according to the current load of the target node's storage IO, adjust the data sending rate in real time.

[0070] S3: Incremental data capture and multi-level caching (source side):

[0071] Full-scale stage:

[0072] The source-side EMA reads source database data shard by shard based on the MCC sharding plan, prioritizing primary key scans or efficient index scans.

[0073] Use progressive reading: Instead of reading the entire shard into memory at once, read it in batches (for example, 1,000 rows at a time), processing and transmitting while reading, greatly reducing memory consumption.

[0074] Incremental phase:

[0075] When the full migration begins, the source EMA starts low-overhead CDC. Capture mechanism selection:

[0076] First choice: Leverage the database's native transaction log (such as MySQL Binlog, PostgreSQL WAL, or Oracle Redo Log). EMA parses the log to extract the data changes (Insert, Update, Delete) relevant to the migration scope and their context (transaction ID, timestamp / SCN).

[0077] Alternative: If native log access is limited or the overhead is too high, use triggers or timestamps / version numbers to track changes (less efficient, but a fallback).

[0078] Multi-level change cache:

[0079] In-memory queue: Parsed change events are first stored in an efficient in-memory queue (such as the Disruptor RingBuffer). The queue is strictly sorted by transaction ID and timestamp.

[0080] Persistence Queue: When the memory queue reaches a threshold (based on memory limits or network conditions), older, untransmitted change events are asynchronously persisted to the source node's local disk (using an embedded KV store such as RocksDB or LevelDB). This ensures that changes are not lost when memory is low or the EMA is restarted.

[0081] Metadata management: Maintains a lightweight global change bitmap, or a Cuckoo Filter variant, that records which data pages or row ranges have changed since a certain point in time. This structure has a minimal memory footprint and is used to quickly locate changes during subsequent incremental merges.

[0082] S4: Efficient data transmission mechanism

[0083] 1. Parallel multi-channel transmission:

[0084] The source-side EMA establishes multiple independent transmission channels (based on TCP or QUIC) in parallel for different data shards or CDC change batches based on the MCC's routing planning strategy.

[0085] Each channel can be bound to a specific network path (e.g., primary path, backup path 1, backup path 2), fully utilizing the bandwidth resources of multiple paths. The actual number of channels that can be established is limited by the available node resources and the global scheduling of the MCC.

[0086] 2. Three-layer adaptive compression:

[0087] Before data transmission, the source EMA will compress the data blocks (formed by data fragments or CDC change batches), significantly reducing network transmission volume. Compression adopts a three-layer adaptive strategy:

[0088] Algorithm selection layer: Dynamically selects the optimal compression algorithm based on the data type (structured data, text, binary LOB) and real-time CPU load. For example, for text / structured data, the high compression ratio Zstandard (Zstd) is used under low CPU load, while the low-overhead LZ4 is used under high CPU load. For binary LOBs, LZ4 or Zstd is selected after evaluating their compressibility.

[0089] Compression level: After selecting an algorithm, the compression level (e.g., Zstd levels 1-19) is dynamically adjusted based on current network bandwidth utilization and the remaining transfer time window. A high level (pursuing a high compression ratio) is used when the network is idle or time is ample; a low level (reducing CPU overhead and accelerating transfer) is used when the network is congested or time is tight.

[0090] Block size layer: Dynamically adjusts the size of pre-compressed data blocks based on network RTT (round-trip time) and packet loss rate. In high RTT / high packet loss environments, the block size is reduced to reduce retransmission costs; in low RTT / low packet loss environments, the block size is increased to improve overall throughput.

[0091] 3. Reliable transmission and breakpoint resumable transmission:

[0092] The transmission channel is based on the QUIC protocol or TCP with the BBR congestion control algorithm, which optimizes congestion control to adapt to fluctuations in the edge network.

[0093] Within each transmission channel, a block confirmation and selective retransmission mechanism is implemented: each data block sent has a unique sequence number and requires confirmation by the receiver; only lost or damaged data blocks need to be retransmitted, rather than entire data slices or change batches.

[0094] Both the source and target EMAs persistently record the block transfer status (e.g., sent, confirmed, pending retransmission) of each data shard / CDC batch. This ensures that even after an EMA process restart, the transfer can be resumed from the exact breakpoint.

[0095] 4. Bandwidth shaping and priority scheduling:

[0096] The source-side EMA implements bandwidth shaping on each transmission channel based on the MCC's global scheduling strategy and real-time network monitoring data to prevent a single channel or total traffic from occupying the bandwidth and affecting normal business.

[0097] At the same time, a higher transmission priority is set for CDC change data than for full data to ensure that incremental changes can be synchronized to the target end as quickly as possible, thereby effectively shortening the time required for the final switch.

[0098] S5: Target-side data writing and consistency assurance

[0099] 1. Full data writing:

[0100] The target EMA receives the chunked data from the source.

[0101] First, decompress the data. Then, use efficient batch insert methods (such as MySQL's LOADDATA INFILE or PostgreSQL's COPY command) to write the data to the target database. This approach significantly reduces the transaction overhead associated with single inserts and improves write performance. Before writing, perform necessary data type conversion or cleaning operations based on predefined rules.

[0102] 2. Incremental change application:

[0103] The target-side EMA receives and decompresses incremental change batches (CDC changes) from the source. In-memory transaction queue: Received change events are first queued in memory and strictly sorted by transaction ID and timestamp, ensuring that changes are processed in the correct order. Idempotent application: When applying changes to the target database, the operation is guaranteed to be idempotent. This means that even if the same change event is received and processed multiple times due to network retransmissions or other factors, the final result is guaranteed to be correct. This is achieved by using statements such as INSERT ... ON DUPLICATE KEY UPDATE (update if exists, insert if not). The database log's precise row location capabilities (for example, configuring binlog_row_image=FULL in MySQL) ensure accurate target row location and update. Transaction consistency: The target-side EMA ensures transaction atomicity. This means that all changes belonging to the same source transaction are either successfully applied to the target database or rolled back if they fail, preventing partial application. To track progress, EMA maintains an apply bitmap that records which changes at which transaction log positions (such as LSN - Log Sequence Number or SCN - System Change Number) have been successfully applied.

[0104] 3. Target-side resource buffering:

[0105] When the target database faces excessive write pressure, the target-side EMA temporarily buffers pending change events in a local persistence queue. This buffering mechanism effectively smooths the write load, preventing sudden traffic from overwhelming the target database and ensuring the stability of the synchronization process and the availability of the target database.

[0106] S6: Migration Status Monitoring, Verification, and Switching

[0107] Real-time monitoring: The real-time dashboards provided by MCC and EMA provide comprehensive monitoring of the migration status, including migration progress (such as the number of completed shards and CDC latency), system resource consumption (CPU, memory, network I / O, disk I / O), network connection quality, and error log information.

[0108] Data Verification: Shard-level Verification: After completing the full data migration for each shard, the source-side EMA calculates a compact checksum for that shard (using a low-CPU-cost algorithm such as XXH3 or CityHash) and sends the result to the target-side EMA. After the data is written, the target-side EMA calculates the checksum for the same shard for comparison and verification.

[0109] Incremental checkpointing: During the incremental data catch-up phase, checkpointing is performed periodically (e.g., every hour) or on-demand. This includes row-level sampling checkpointing or fast change summary comparison using previously built GCB / Cuckoo Filter. Source and target compute hash summaries for a specified range (row or page) and compare the results.

[0110] Final consistency checkpointing: Before the formal switchover, a more comprehensive (possibly sampling-based) data consistency check is performed as the final confirmation.

[0111] Near-zero downtime switchover: Suspend source database writes: Choose a period of low business traffic and coordinate the application to temporarily stop (usually seconds) writing to the source database. Catch up the last increment: The source EMA captures the last batch of incremental change data generated during the suspension period and transmits it to the target. The target EMA quickly applies this batch of changes. Final checkpoint confirmation: Perform a quick final data checkpoint to ensure that the target database data is up-to-date and consistent. Switch application traffic: Update configurations (such as DNS, load balancer policies, application connection strings) to direct the application's read and write traffic to the target database. Restore source database reads: If the source database is planned to be retained as a standby or read-only database, its read traffic can be restored at this time.

[0112] Rollback mechanism: If serious problems occur after switchover, provide a solution for quick rollback to the source database (this solution usually relies on the forward / backward compatibility design of the application layer).

[0113] S7: Resource recycling and cleaning:

[0114] After confirming that the migration is successful and stable: Stop the EMA processes on the source and target (or put them into idle mode). Clean up the source and target persistent caches, queues, and state information. Release related resources.

[0115] An edge computing-based database rapid migration system for implementing the above method, comprising:

[0116] Migration Coordination Center (MCC): Deployed in the edge cloud or central cloud. The Migration Coordination Center (MCC) includes a task management module: receives, parses, and manages migration task requests. Topology discovery and monitoring module: collects, stores, and analyzes node resources, network paths, and database information reported by EMA. Intelligent planning engine: executes data sharding strategies, runs dynamic path optimization algorithms, and generates migration plans (sharding, routing, and resource quotas). Global scheduler: sends the migration plan to EMA, coordinates the parallel work of multiple EMAs, processes EMA's heartbeat and status reports, and makes global load balancing and failover decisions. Monitoring and Alarm Center: provides a visual dashboard to display global migration status, performance indicators, and alarm information. API Gateway: provides a RESTful API for users to create, manage, and monitor tasks.

[0117] It also includes the Edge Migration Agent (EMA): a lightweight software agent deployed on the source and target edge nodes.

[0118] Furthermore, the Edge Migration Agent (EMA) includes local resource and network probes: it collects information on the resource utilization of the local node and the network quality between the local node and other related nodes. The database connector connects to the source or target database and performs data reads (full data), log parsing (CDC), and data writes (batch / change application). It supports multiple database drivers. The data shard processor (source side) receives sharding instructions from the MCC and performs specific data shard read operations (progressive acquisition). The change data capture (CDC) engine (source side) captures database changes with low overhead, manages multi-level change caches (memory queues and persistent queues), and maintains GCB / Cuckoo filters. The transmission engine (source side) establishes and maintains multiple transmission channels. The compression engine performs three-layer adaptive compression. The reliable transmission controller implements block confirmation, selective retransmission, state persistence, and breakpoint resuming. The traffic shaper controls the sending rate and priority based on the scheduling policy. The receiving and decompression engine (target side) receives data blocks and decompresses them. Data Write Engine (Target): Performs batch inserts (full) and idempotent change applications (incremental), managing the target-side buffer queue. Local State Manager: Persistently stores information such as shard transfer status, change application status, and checksums, ensuring state recovery after EMA restarts. Heartbeat and Reporting Module: Periodically sends heartbeats and local status (progress, resources, and errors) to the MCC.

[0119] Furthermore, relay edge nodes are also included: when the source and destination cannot be directly connected or the direct connection quality is poor, EMA is deployed as a relay node, responsible for receiving, temporarily storing (optionally), and forwarding data. Relay nodes also participate in MCC scheduling and path optimization.

[0120] Example 2, migrating a MySQL database from edge node A to edge node B.

[0121] Environment deployment:

[0122] Deploy the MCC instance in the regional edge cloud.

[0123] Install the lightweight EMA package on edge node A (source) and edge node B (target).

[0124] Task initialization and topology awareness (S1):

[0125] The user submits a migration task through the MCC API: source = node A MySQL (IP:Port, User / PW), target = node B MySQL (IP:Port, User / PW), migration range = entire mydb database.

[0126] EMA-A starts and probes node A: 4-core ARM CPU, 2GB available memory, eMMC storage, connected to gateway GW1. The node B path is detected: 80ms ± 20ms delay through GW1, available uplink bandwidth 5Mbps (bottleneck), packet loss rate 1%.

[0127] EMA-B starts and probes node B: similar resources. The node A path is the same. Information is reported to the MCC.

[0128] EMA-A connects to the source MySQL and obtains all table structures and table size estimates in information_schema.

[0129] Smart sharding and routing planning (S2):

[0130] MCC analysis: large table orders (estimated 100GB) is the bottleneck, using primary key range sharding, network bandwidth is the main limitation (5Mbps uplink), target node B storage IOPS is general.

[0131] Sharding: divide the orders table into 200 shards according to the primary key order_id (about 500MB each). Other small tables are each divided into 1 shard.

[0132] Routing planning: the dynamic path optimization algorithm evaluates that the bandwidth utilization of the current path (A->GW1->B) has reached about 90%, and the RTT has risen to about 120ms. The MCC decides: 1) limit the number of parallel transmission channels to 2 (to avoid excessive congestion); 2) set a bandwidth upper limit of 2Mbps for each channel (total 4Mbps < 5Mbps available); 3) use medium compression level (Zstd level 9).

[0133] The plan is issued to EMA-A and EMA-B.

[0134] Incremental capture and full transfer start (S3, S4, S5):

[0135] CDC Startup (EMA-A): Binlog parsing (using the mysqlbinlog emulation or library interface) is immediately started, capturing all changes to the mydb database. Change events are stored in an in-memory queue (capacity: 10,000 events). A background thread asynchronously persists overflow events to the local RocksDB. The in-memory Cuckoo Filter record of the changes is updated.

[0136] Full transfer (EMA-A & EMA-B):

[0137] EMA-A selects the first shard (orders ID 1-500000), starts thread 1, and reads it incrementally (1000 rows at a time).

[0138] The read data block (approximately 1MB) is fed into the compression module. The current CPU load is moderate (approximately 60%). Zstd and level 9 (as specified by MCC) are selected. The compression ratio is approximately 4:1. The compressed data size is approximately 250KB.

[0139] The transport engine sends the compressed blocks through channel 1. Channel 1 is bound to the path A->GW1->B and has a speed limit of 2 Mbps.

[0140] At the same time, thread 2 is started to process the second shard and send it through channel 2 (also limited to 2Mbps).

[0141] EMA-B receives channel 1 / 2 data, decompresses it (Zstd), and writes it in batches to the target MySQL (LOAD DATAINFILE).

[0142] Each time a complete shard is successfully written, EMA-B calculates the XXH3 checksum and sends it to EMA-A for verification. Once it passes, the shard is marked complete.

[0143] Dynamic adaptation (ongoing):

[0144] Network fluctuations: MCC monitoring detected a sudden increase in the packet loss rate to 5%. This triggered the dynamic path optimization algorithm: 1) EMA-A was notified to reduce the transmission block size (from 1MB to 512KB); 2) the compression level was temporarily lowered (Zstd level 9 to level 5) to reduce CPU overhead associated with retransmissions; and 3) an attempt was made to detect a more optimal path (e.g., through another gateway, GW2, which has slightly higher latency but lower packet loss).

[0145] Resource Insufficiency: EMA-A monitors a CPU load of 85%. Local strategy (or report to the MCC): 1) Temporarily reduce the compression level of non-critical shards (for example, use LZ4 for small tables); 2) Slightly reduce the priority of the CDC memory queue.

[0146] Segment completion: When a segment is completed, EMA-A notifies the MCC. The MCC scheduler assigns the next segment to be migrated to an idle or lightly loaded transmission thread / channel.

[0147] Incremental data transmission and application (S4, S5):

[0148] While the full data is being transferred, EMA-A continues to capture changes.

[0149] The transfer engine creates a high-priority channel for CDC data (or reuses an existing channel but marks it as high priority). CDC change batches are compressed (usually using LZ4 to ensure speed) before transmission.

[0150] EMA-B receives CDC data, decompresses it, sorts it by transaction ID, and applies it to the target database in an idempotent manner (using INSERT ... ON DUPLICATE KEY UPDATE ...). It also records the applied Binlog position.

[0151] Verification and switching (S6):

[0152] Full completion: All 200 orders shards and other table shards have been transferred and verified.

[0153] Incremental catching up: Before EMA-A pauses writing, CDC latency stabilizes within 5 seconds. MCC triggers switchover preparation.

[0154] Pause source writes and catch up with the last increment: Coordinate the application to pause writes to the source database (taking < 1 second). EMA-A captures and transmits the changes from the last few seconds. EMA-B applies the changes in < 1 second. A quick checksum of several key tables affected by the last changes is performed.

[0155] Switchover: Update the application configuration center to redirect the mydb connection to MySQL on Node B (takes < 1 second). Resume application write operations. Total downtime ≈ 2 seconds.

[0156] Rollback plan: If a problem is found after the switch, the connection can be quickly switched back to node A (using the consistency point at the previous pause).

[0157] Cleaning (S7):

[0158] After confirming that MySQL on node B is running stably for an hour, the MCC notifies EMA-A and EMA-B to stop the task. EMA-A cleans up the local RocksDB CDC cache and transmission status, and EMA-B cleans up the local status.

[0159] Close the associated connection.

[0160] The edge computing-based rapid database migration method and system designed in this invention can be widely applied to: Internet of Things (IoT) platforms: sinking device management and time-series data storage from the cloud to edge gateways or regional edge nodes. Industrial Internet: migrating real-time production databases between different production line control servers within a factory to support production line adjustments or disaster recovery. Smart retail / chain stores: migrating and synchronizing store sales and inventory databases on demand between the headquarters cloud, regional distribution center edge, and store edge servers. Internet of Vehicles (V2X): migrating vehicle information and road condition information databases between roadside units (RSUs) or regional MEC nodes. Smart cities: migrating security monitoring, traffic flow, and environmental monitoring databases between street-level, district-level, and city-level edge nodes. CDN and edge video: migrating user configurations and content metadata to edge POPs closer to users. Telecom 5G MEC: migrating user plane function (UPF)-related status data or application databases between UPFs or MEC hosts. This system significantly improves the efficiency, reliability, and user experience of database migration in edge scenarios, and has broad market prospects and extremely high practical value.

[0161] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A database rapid migration method based on edge computing, characterized in that: The steps include: S1: Receives a migration task request, deploys a source edge migration agent (EMA) on the source edge node, deploys a target EMA on the target edge node, performs edge topology awareness, collects information, and reports it to the migration coordination center (MCC). S2: The MCC performs intelligent data sharding to generate multiple data slices. It then runs a dynamic path optimization algorithm to plan the optimal transmission path for each data slice, and formulates a migration plan for distribution. S3: The source-side EMA starts low-overhead change data capture (CDC) based on the migration plan, manages change data using a multi-level cache, and performs progressive reads by data slice. S4: The source-side EMA establishes parallel multi-transmission channels, applies a three-layer adaptive compression strategy to the read data blocks or CDC change batches, and reliably transmits them through the planned optimal transmission path. S5: The target EMA receives and decompresses the data, performs batch writes on the full data, sorts the incremental changes by transaction, and applies them idempotently to the target database. S6: Perform data verification and execute near-zero downtime switchover after the data is consistent; S7: After the migration is completed, resources are recycled and cleaned up.

2. The method for rapid database migration based on edge computing according to claim 1, characterized in that: The following steps are also included: The edge topology perception includes: detecting the CPU, memory, disk performance and capacity information of the source edge node and the target edge node; detecting the direct or indirect network path between the source edge node and the target edge node; and obtaining the source database and target database information.

3. A database rapid migration method based on edge computing according to claim 1 or 2, characterized in that: The following steps are also included: The dynamic path optimization algorithm is a model or heuristic algorithm based on reinforcement learning (Q-Learning). It continuously evaluates path status and node load and dynamically adjusts the routing of data slices. The adjustment strategies include: switching to a backup path when the primary path is congested; adjusting the sending rate based on the target node storage IO load; and limiting the number of parallel transmission channels.

4. The method for rapid database migration based on edge computing according to claim 1, characterized in that: The low-overhead CDC includes: prioritizing parsing database transaction logs to capture changes; first storing change events in a memory queue sorted by transaction ID and timestamp; when the memory queue reaches a threshold, asynchronously persisting older untransmitted events to local storage; and maintaining a global change bitmap or Cuckoo Filter variant to record the range of changed data.

5. A database rapid migration method based on edge computing according to claim 1 or 4, characterized in that: The reliable transmission includes: using the QUIC protocol or TCP with BBR to optimize congestion control; implementing block confirmation and selective retransmission; persistently recording the block transmission status of each data slice / CDC batch to achieve breakpoint resumption; setting a higher transmission priority for CDC change data than the full data; and performing bandwidth shaping according to the MCC policy.

6. The method for rapid database migration based on edge computing according to claim 1, characterized in that: The three-layer adaptive compression strategy is specifically: Algorithm selection layer: The Zstandard algorithm is used for text / structured data when the CPU load is low, and the LZ4 algorithm is used when the CPU load is high. Binary LOBs are selectively compressed after evaluation. Compression level tier: Use high compression level when the network is idle or there is ample migration time, and use low compression level when the network is congested or time is tight; Block size layer: Reduce the pre-compression data block size when RTT / packet loss is high, and increase the block size when RTT / packet loss is low.

7. The method for rapid database migration based on edge computing according to claim 1, characterized in that: The idempotent application of incremental change data includes: using an INSERT ... ON DUPLICATE KEY UPDATE statement or precise positioning based on a complete row image; ensuring atomic application of changes within transaction boundaries; and maintaining an application bitmap to record transaction log positions that have been successfully applied.

8. The method for rapid database migration based on edge computing according to claim 1, characterized in that: The data verification includes: shard-level compact verification and XXH3, CityHash comparison after each shard is completed in the full migration; row-level sampling verification or change summary comparison based on the global change bitmap / Cuckoo Filter during the incremental migration phase; and final consistency check before switching.

9. The method for rapid database migration based on edge computing according to claim 1, characterized in that: The near-zero downtime switch includes: pausing write operations on the source database; transmitting and applying the last incremental changes generated during the pause; performing a fast final check; switching application traffic to the target database; and providing a fast rollback mechanism designed based on the application layer.

10. A database rapid migration system based on edge computing, characterized in that: Used to implement the method according to any one of claims 1 to 9, comprising: Migration Coordination Center (MCC): Deployed in the edge cloud or central cloud; includes a task management module, topology discovery and monitoring module, intelligent planning engine, global scheduler, monitoring and alarm center, and API gateway. It also includes the edge migration agent EMA: a lightweight software agent deployed on the source edge node and the target edge node; including local resource and network probes, database connectors, source-side data sharding processors, source-side change data capture CDC engine, source-side transmission engine, reliable transmission controller, traffic shaper, target-side receiving and decompression engine, target-side data writing engine, local state manager, heartbeat and reporting module.

Images