A breakpoint resuming migration method, device and medium for distributed migration of a heterogeneous database

Abstract

The embodiment of the application discloses a breakpoint resuming migration method, equipment and medium for heterogeneous database distributed migration, belongs to the technical field of heterogeneous database migration, and solves the problem that after the abnormal termination of heterogeneous database migration, the whole migration process is usually restarted, computing resources are wasted, and the total migration time is prolonged. The method comprises the following steps: a system service node persists migration execution information, and a migration service node persists execution task information; in response to the abnormal interruption of the system service node and / or the migration service node, after recovery, the system service node determines completed execution tasks and uncompleted execution tasks based on the persisted migration execution information; a breakpoint resuming migration process is started, the system service node reassigns the determined uncompleted execution tasks to the migration service node; and the migration service node checks whether the uncompleted execution tasks have been completed based on the execution task information, and returns the execution result to the system service node.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of heterogeneous database migration, and particularly relates to a heterogeneous database distributed migration breakpoint resuming method, device and medium. BACKGROUND

[0002] With the deepening of enterprise digital transformation, the demand for migrating business systems from one database platform to another heterogeneous database platform is increasing. Such heterogeneous database migration involves the conversion of different data structures, syntax and storage engines, and is usually a complex and time-consuming system engineering. When migrating massive data or processing complex business logic, the migration task often needs to run for a long time.

[0003] The existing heterogeneous database migration tools mainly focus on data format conversion, structure mapping and basic data synchronization. However, in actual production environment, the migration process often faces the risk of interruption due to various external uncontrollable factors, such as network connection flash-off, response timeout caused by excessive pressure on the source or target database system, hardware failure or operation interference, etc.

[0004] Once the current mainstream migration tool terminates abnormally in the execution process due to any of the above unexpected reasons, the operator usually restarts the entire migration process from the task starting point, which not only causes the successfully migrated data and objects to be repeatedly processed, resulting in serious waste of computing resources, storage I / O and network bandwidth, but also greatly prolongs the total migration time window, affecting the business switching plan. SUMMARY

[0005] The embodiments of the present application provide a heterogeneous database distributed migration breakpoint resuming method, device and medium, which are used to solve the following technical problems: current heterogeneous database migration, once migration is terminated abnormally, the entire migration process is usually restarted from the task starting point, which not only causes the successfully migrated data and objects to be repeatedly processed, resulting in serious waste of computing resources, storage I / O and network bandwidth, but also greatly prolongs the total migration time window, affecting the business switching plan.

[0006] The embodiment of the present application adopts the following technical scheme: the embodiment of the present application provides a breakpoint resuming migration method for heterogeneous database distributed migration, which is applied to a distributed migration architecture including at least one system service node and at least one migration service node, and the method includes: the system service node persisting migration execution information, and the migration service node receiving a task persisting execution task information; wherein the migration execution information at least includes one of migration execution stage information and execution task information allocated to the migration service node; in response to abnormal interruption of the system service node and / or the migration service node, after recovery of the abnormally interrupted node, the system service node determines completed execution tasks and uncompleted execution tasks based on the persisted migration execution information; a breakpoint resuming migration process is started, and the system service node re-allocates the determined uncompleted execution tasks to the migration service node; the migration service node checks whether the uncompleted execution tasks have been completed based on the execution task information, if the uncompleted execution tasks have been completed, the migration service node returns execution results to the system service node, and if the uncompleted execution tasks have not been completed, the migration service node executes the tasks and returns execution results to the system service node.

[0007] In an implementation manner of the present application, in response to abnormal interruption of the system service node and / or the migration service node, after recovery of the abnormally interrupted node, the system service node determines completed execution tasks and uncompleted execution tasks based on the persisted migration execution information, and specifically includes: when the system service node is abnormally interrupted and the migration service node is normally running or is abnormally interrupted, the migration service node continues to execute tasks and saves results, and after the system service node is restarted, the system service node is restored to an execution stage before the abnormal interruption, and based on the migration execution information, the system service node acquires unallocated tasks in the stage and tasks that have been allocated but not reported to be completed, and re-allocates the acquired tasks; when at least one migration service node is abnormally interrupted and the system service node is normally running, the system service node stops allocating new tasks to the abnormally interrupted migration service node, and marks tasks allocated to the abnormally interrupted migration service node as a timeout unknown state, and when the abnormally interrupted migration service node is online again, the system service node queries related task results from the online migration service node and updates a task state.

[0008] In an implementation manner of the present application, the system service node re-allocates the determined uncompleted execution tasks to the migration service node, and specifically includes: the system service node acquires a list of to-be-allocated migration service nodes according to heartbeat information of the migration service nodes; the system service node allocates the uncompleted execution tasks to nodes in the list of to-be-allocated migration service nodes according to a preset weighted round robin algorithm; and when the tasks are allocated, weights of the nodes are released or allocated based on the number of tasks allocated to the nodes.

[0009] In one implementation of this application, before assigning unfinished execution tasks to nodes in the list of migration service nodes to be assigned, the method further includes: the system service node initiating a global consistency snapshot query request to the target database to obtain a target-side resource status view corresponding to the target database; wherein, the target-side resource status view consists of the structure definition, data distribution overview, and list of unfinished transaction IDs of objects related to the current migration in the target database; the system service node analyzes the unfinished execution tasks to determine conflicting task groups; wherein, the conflicting task groups include at least one of a data conflict group and a structure conflict group; for each unfinished execution task, the system service node simulates the changes to the target database state after executing the unfinished execution task based on the target-side resource status view, and generates a change version number based on the changes; the system service node generates a conflict-free scheduling sequence corresponding to the unfinished execution task based on the changes to the target database state and the conflicting task groups, and assigns the conflict-free scheduling sequence to the corresponding nodes in the list of migration service nodes to be assigned.

[0010] In one implementation of this application, after allocating the conflict-free scheduling sequence to the corresponding node in the list of migration service nodes to be allocated, the method further includes: the system service node sending the version change number related to the incomplete execution task to the corresponding node in the list of migration service nodes to be allocated; before executing the task, the migration service node determines the snapshot version information used by the system service node for inference based on the version change number, and executes the task and reports the task success to the system service node when the current state of the relevant object in the target database is consistent with the snapshot version information; after receiving the task success report from any migration service node, the system service node updates the global target resource status view, increments the relevant global version number, and broadcasts the global version number as an event to all online migration service nodes.

[0011] In one implementation of this application, after the system service node determines the completed and incomplete execution tasks based on persistent migration execution information, the method further includes: if the system service node detects an abnormal interruption, and any shard migration task belonging to the same table is not completed, the system service node performs resharding on all data of the table; based on the result of the resharding, a new shard migration task is generated and allocated.

[0012] In one implementation of this application, the system service node performs resharding on all data in the table, specifically including: the system service node determining the range of data that has been successfully migrated to the target table before the abnormal interruption, and marking the verified continuous data intervals as safe intervals; based on the safe intervals, the system service node determining the data gaps in the entire table that have not been migrated and whose status is unknown; based on the processing capability status of the migration service node to be assigned the task and the data characteristics corresponding to the data gaps, the system service node dynamically generates corresponding data shards to be migrated for each data gap; wherein, the data characteristics include at least one of primary key dispersion and data volume; the system service node performs boundary alignment operation on the data shards to be migrated, so that the boundaries of the data shards to be migrated are aligned with both the boundaries of adjacent safe intervals and the physical storage unit boundaries of the target database.

[0013] In one implementation of this application, after the system service node reallocates the identified incomplete execution tasks to the migration service node, the method further includes: dividing the migration process into multiple migration stages with dependencies; constructing a graph model reflecting the dependencies between tasks based on the metadata of the objects to be migrated; updating the graph model according to the persistent task completion status, and dynamically determining the set of tasks to be scheduled corresponding to the current migration stage based on the updated graph model status; scheduling tasks sequentially in the set of tasks to be scheduled according to the dependency order of the migration stages, and after all execution tasks in the current migration stage have been migrated, proceeding to schedule the execution tasks of the next migration stage.

[0014] This application provides a breakpoint resume device for distributed migration of heterogeneous databases, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: persist migration execution information by a system service node, and persist execution task information by a migration service node receiving the task; wherein the migration execution information includes at least one of migration execution stage information and execution task information allocated to the migration service node; responding to abnormal interruption of the system service node and / or the migration service node, after the abnormal interruption node recovers, the system service node determines the completed and uncompleted execution tasks based on the persisted migration execution information; initiates the breakpoint resume process, the system service node reallocates the determined uncompleted execution tasks to the migration service node; the migration service node checks whether it has completed the uncompleted execution tasks based on the execution task information, if completed, returns the execution result to the system service node, if not completed, executes the task and returns the execution result to the system service node.

[0015] This application provides a non-volatile computer storage medium storing computer-executable instructions. These instructions are configured to: persist migration execution information by a system service node; and persist execution task information by a migration service node receiving a task. The migration execution information includes at least one of migration execution stage information and execution task information allocated to the migration service node. In response to an abnormal interruption of the system service node and / or the migration service node, after the abnormally interrupted node recovers, the system service node determines completed and incomplete execution tasks based on the persisted migration execution information. A breakpoint resume process is initiated, and the system service node reallocates the determined incomplete execution tasks to the migration service node. Based on the execution task information, the migration service node checks whether it has completed any incomplete execution tasks. If completed, it returns the execution result to the system service node; if incomplete, it executes the task and returns the execution result to the system service node.

[0016] The above-mentioned technical solutions adopted in this application embodiment can achieve the following beneficial effects: Firstly, by using distributed collaborative persistence, this application embodiment disperses the system state across multiple nodes, avoiding the risk of total state loss due to a single node failure and ensuring the recoverability of the entire migration system state. Secondly, after the abnormally interrupted node recovers, this application embodiment utilizes the previously persisted information to perform intelligent state reconstruction by the system service node, accurately distinguishing between completed and uncompleted tasks. This allows the system to precisely locate the exact progress point at the time of the interruption, enabling it to start from the breakpoint rather than from zero, avoiding the huge resource waste and time loss caused by a full restart in traditional solutions. Furthermore, this application embodiment re-injects uncompleted execution tasks into the running distributed workflow, achieving automation of the recovery process and optimization of resource rescheduling, optimizing the overall utilization of cluster computing resources, and ensuring that the continued migration task can be executed efficiently and smoothly. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0018] Figure 1 A flowchart of a breakpoint resume method for distributed migration of heterogeneous databases provided in this application embodiment;

[0019] Figure 2 This application provides a schematic diagram of the stages and task allocation process during system service migration.

[0020] Figure 3 This is a schematic diagram illustrating the process of executing a migration task using a migration service, as provided in an embodiment of this application.

[0021] Figure 4 A schematic diagram of a system service breakpoint resume process provided in this application embodiment;

[0022] Figure 5 This is a schematic diagram of a breakpoint resume device for distributed migration of heterogeneous databases, provided in an embodiment of this application.

[0023] Figure label:

[0024] 200: Breakpoint resume device for distributed migration of heterogeneous databases; 201: Processor; 202: Memory. Detailed Implementation

[0025] This application provides a method, device, and medium for resuming interrupted migrations in a distributed migration of heterogeneous databases.

[0026] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0027] Figure 1 A flowchart of a breakpoint-resuming method for distributed migration of heterogeneous databases provided in this application embodiment is shown below. Figure 1 As shown, the breakpoint resume method for distributed migration of heterogeneous databases includes the following steps:

[0028] Step 101: The system service node persists the migration execution information, and the migration service node receiving the task persists the execution task information.

[0029] In one implementation of this application, when migrating between heterogeneous databases using a distributed architecture, for migration tasks interrupted due to a migration tool service crash, a resume operation can be performed after the migration tool restarts. The distributed migration architecture consists of a system service and a migration service. The system service is responsible for task scheduling, while the migration service is responsible for task execution. The system service simultaneously receives the results of the migration service's task execution and persists the current execution stage and the tasks already assigned to the migration service. After a task is completed, its status is updated in the JVM (Java Virtual Machine) memory and the database. This ensures that even in the event of an abnormal interruption, it is clear which tasks have been completed and which tasks, while not completed, have been assigned to the migration service. When the migration service receives a migration task to be executed, it persists the task locally for confirmation of its completion status after the migration tool restarts. The migration execution information includes at least one of the following: migration execution stage information and information about the execution tasks assigned to the migration service nodes.

[0030] Step 102: Respond to abnormal interruptions of the system service node and / or migration service node. After the abnormal interruption node recovers, the system service node determines the completed and uncompleted execution tasks based on the persistent migration execution information.

[0031] In one implementation of this application, when a system service node experiences an abnormal interruption, and the migration service node is operating normally or experiencing an abnormal interruption, the migration service node continues to execute tasks and saves the results. After the system service node restarts, it resumes the execution phase before the abnormal interruption and, based on the migration execution information, retrieves unassigned tasks and assigned but not yet reported completed tasks from that phase, and then reallocates the retrieved tasks. When at least one migration service node experiences an abnormal interruption while the system service node is operating normally, the system service node stops assigning new tasks to the interrupted migration service node and marks the tasks already assigned to the interrupted migration service node as having a timeout unknown status. When the interrupted migration service node comes back online, the system service node queries the relevant task results from the re-online node and updates the task status.

[0032] Specifically, the embodiments of this application address the scenario of abnormal interruption during migration in a distributed architecture (single system service + multiple migration services) as follows:

[0033] (1) System service is abnormally interrupted, but migration service is running normally;

[0034] (2) System services and all migration services were abnormally interrupted;

[0035] The steps for resuming the migration process corresponding to (1) and (2) above are as follows: After the system service is abnormally interrupted, the migration service continues to perform the migration and saves the execution results. After the system service restarts, it returns to the stage at the time of the abnormal interruption and obtains the tasks that have not been assigned to be executed and the tasks that have been assigned (for the assigned tasks, it records which migration service node the task was assigned to run). These two types of tasks are reassigned to the migration service. The migration service checks whether the task has been completed. If it has been completed, the execution result is returned directly to the system service. If the task has not been executed, it will be executed first and then the result will be returned to the system service.

[0036] (3) System services are running normally + 1 migration service is interrupted abnormally + other migration services are running normally;

[0037] The above (3) corresponds to the breakpoint resume operation steps: After the system service heartbeat detects that the migration service A has gone offline, it will no longer assign tasks to that service node, but instead assign tasks to other service nodes. For the assigned tasks, the task status can be changed to unknown by the set timeout period (the timeout period is when the migration service node has not restarted or come back online within a certain period of time). When the migration service node comes back online, the system service will send the ID of the executed migration task to the migration service node. The migration service node will return the result of the executed task corresponding to the ID to the system service node, which will then update the system service. At the same time, the system service will assign a new migration task to the node.

[0038] Step 103: Start the breakpoint resume process. The system service node will reassign the identified unfinished execution tasks to the migration service node.

[0039] In one implementation of this application, the system service node obtains a list of migration service nodes to be assigned based on heartbeat information with each migration service node. The system service node then assigns incomplete execution tasks to nodes in the list of migration service nodes to be assigned according to a pre-set weighted round-robin algorithm. Furthermore, during task assignment, the system service node releases or allocates weight tokens to each node based on the number of tasks assigned to each node.

[0040] Specifically, in this embodiment, the system service obtains a list of all available migration service nodes through a heartbeat mechanism with the migration service. Users can also select the migration service node to use when starting a migration. Each migration service node can be configured with its own weight, for example, m1=64, m2=8 (configured according to hardware conditions). This weight represents the maximum number of tasks that can be executed simultaneously during task allocation. When the system service starts a migration, it obtains available migration service nodes according to a weighted round-robin algorithm to ensure even allocation. It also checks whether the node's weight is available; for example, the system service might allocate 64 tasks to m1 and 8 tasks to m2. Each time m1 completes a task, its weight token recorded in the system service is released by 1, allowing it to be allocated a new task. Furthermore, the list of available migration services maintained by the system service in this embodiment can be dynamically modified. If a migration service disconnects due to a heartbeat timeout, it will be removed from the context of all running migration tasks, and tasks will not be allocated to that node during reassignment.

[0041] In one implementation of this application, before assigning incomplete execution tasks to nodes in the list of migration service nodes to be assigned, the system service node further includes initiating a global consistency snapshot query request to the target database to obtain a target-side resource status view corresponding to the target database. The target-side resource status view consists of the structure definition, data distribution overview, and list of incomplete transaction IDs of objects related to the current migration in the target database. The system service node analyzes the incomplete execution tasks and identifies conflicting task groups. These conflicting task groups include at least one of a data conflict group and a structure conflict group. For each incomplete execution task, the system service node simulates the changes to the target database state after executing the incomplete execution task in a pre-defined target-side sandbox simulation layer based on the target-side resource status view, and generates a change version number based on these changes. Based on the changes to the target database state and the conflicting task groups, the system service node generates a conflict-free scheduling sequence corresponding to the incomplete execution task, and assigns the conflict-free scheduling sequence to the corresponding nodes in the list of migration service nodes to be assigned.

[0042] Specifically, in this embodiment of the application, before initiating breakpoint resume and reassigning tasks, the system service node proactively initiates a global consistency snapshot query request to the target database to obtain a view of the target database's resource status at the current moment. To achieve this, the system service node sends a specific query command to the target database; for example, in databases that support snapshot isolation levels, it queries system tables or obtains relevant metadata through the database management interface. The obtained status view should at least include the current structure definition of all database objects related to this migration, an overview of the data distribution of core tables, and a list of currently active uncommitted transaction IDs.

[0043] Next, the system service node performs static analysis on all incomplete execution tasks to identify task groups with potential execution conflicts, including data conflicts and structural conflicts. For data conflicts, the analysis examines the data scope operated on by different tasks, such as the primary key range defined by the WHERE condition in an SQL statement, to determine if there is overlap or read-write dependencies. For structural conflicts, the analysis examines whether different tasks involve structural modifications to the same database object. By parsing the semantics of each task, the system service node extracts these key attributes to cluster tasks with overlapping data or object dependencies into different conflict groups.

[0044] Then, the system service node simulates the impact of each incomplete task through a pre-configured target-side sandbox simulation layer. This simulation layer maintains a software abstraction of the target database state in the system service node's memory. The system service node uses the acquired target-side resource state view as the initial state for simulation. For each incomplete task, the system service node simulates the execution of the task internally. For example, simulating the execution of an INSERT statement marks the corresponding data range as occupied in the memory model; simulating the creation of an index adds the index structure to the model and calculates the expected change to the target database state after execution. After each simulation execution, a globally unique, incrementing version number is generated for this change.

[0045] Finally, for all tasks belonging to the same conflict group, they are assigned to the same migration service node and executed serially according to the order in which their version numbers are changed. This ensures the atomicity and consistency of operations on the target side, avoiding deadlocks or inconsistent states. Based on this constraint, the system service node can use a graph scheduling algorithm to sort all tasks, forming a global task execution sequence. After determining this sequence, the system service node then selects tasks sequentially from the sequence based on the currently available migration service node list and its load, deciding which node each task should be assigned to for execution, thus completing the process from conflict analysis to final task allocation and scheduling.

[0046] In one implementation of this application, after selecting nodes to be assigned tasks from the conflict-free scheduling sequence, the system service node further includes sending the version change number related to the incomplete execution task to the corresponding node in the list of migration service nodes to be assigned. Before executing the task, the migration service node determines the snapshot version information used by the system service node during the simulation based on the version change number. When the current state of the relevant object in the target database is consistent with the snapshot version information, the migration service node executes the task and reports a successful task to the system service node. After receiving a successful task report from any migration service node, the system service node updates the global target resource status view, increments the relevant global version number, and broadcasts the global version number as an event to all online migration service nodes.

[0047] Specifically, after the system service node determines the tasks to be assigned and their target nodes based on the conflict-free scheduling sequence, it issues a task execution instruction to the selected migration service node. This instruction not only includes the task itself but also includes a version number generated during the simulation phase related to the task. This version number is a logical marker indicating the expected changes to the global target resource status view after the system service node internally simulates the execution of the task.

[0048] Furthermore, upon receiving a task instruction that includes a version number change, the migration service node, based on this version number, reverse-engineers the snapshot version information used by the system service node during the sandbox simulation—that is, the state identifier of the target database at a certain consistent snapshot point. Subsequently, the node initiates a verification request to the target database. The core of this request is to verify whether the current state of relevant objects in the target database is consistent with the state identified by the derived snapshot version information. Specifically, this can be accomplished by querying the target database's system version number, last modified timestamp, or by querying execution conditions. This step serves as a safeguard mechanism to prevent operation failure or data inconsistency caused by accidental modification of the target's state during scheduling and execution intervals.

[0049] Furthermore, if the verification passes, confirming that the current state of the target endpoint matches the expected baseline state during system service node scheduling, the migration service node begins safely executing the actual migration task assigned to it. Upon successful task completion, the migration service node immediately sends a task completion report message to the system service node, containing the task ID and a summary of the execution result. If the verification fails, the migration service node does not execute the task, returns an outdated state version error code and the currently detected actual state information to the system service node, requesting the system service node to reconsider its decision.

[0050] Furthermore, upon receiving a successful task completion message from any migration service node, the system service node updates its maintained global target resource state view in memory to reflect the state change caused by the task, based on the successful execution of the task. Next, the system service node increments the globally maintained global version number to ensure the integrity of the state changes. Finally, the system service node actively and in real-time pushes this global version number increment event as a broadcast notification to all currently online migration service nodes. Upon receiving this broadcast, each migration service node synchronously updates its locally cached baseline state version information used for subsequent task verification, thereby ensuring that the entire distributed system maintains a consistent understanding of the target state.

[0051] In one implementation of this application, if any shard migration task belonging to the same table is not completed when the system service node detects an abnormal interruption, the system service node performs re-sharding on all data of the table. Based on the result of the re-sharding, a new shard migration task is generated and allocated.

[0052] Specifically, for table data operations, the granularity of breakpoint resume is the entire table. If one shard fails to complete at the breakpoint, the entire table data will be re-migrated. That is, when a table is split into 10 shard tasks and assigned to 10 migration services, if one of the shard tasks executed in a migration service fails to complete, the breakpoint resume will, while the system service checks the task completion status, re-divide the table into 10 shards and reassign them to the migration services for execution. At this time, it is not necessary to reassign the original shards to the original migration services.

[0053] In one implementation of this application, the system service node performs resharding on all data in the table. Specifically, this includes: the system service node determining the range of data successfully migrated to the target table before the abnormal interruption, and marking the verified continuous data intervals as safe intervals. Based on the safe intervals, the system service node determines the data gaps in the entire table that have not been migrated and whose status is unknown. Based on the processing capacity status of the migration service nodes to be assigned tasks and the data characteristics corresponding to the data gaps, the system service node dynamically generates corresponding data shards to be migrated for each data gap; wherein the data characteristics include at least one of primary key dispersion and data volume. The system service node performs boundary alignment operations on the data shards to be migrated, aligning the boundaries of the data shards to be migrated with both the boundaries of adjacent safe intervals and the physical storage unit boundaries of the target database.

[0054] Specifically, when a system service node initiates a resume migration, it first interacts with the target database to accurately identify data that was successfully migrated before the abnormal interruption. Specifically, the system service node sends a query to the target database, comparing the primary key ranges or unique identifier fields of the source and target tables based on persistent task completion records. For each continuous data block confirmed to exist completely in the target table and consistent with the source record, the system service node records it internally and marks it as a safe interval. Each safe interval is defined by a start primary key value and an end primary key value.

[0055] Based on multiple marked safe intervals, the system service node analyzes the logical primary key space of the entire table. All safe intervals are sorted by primary key values, and the intervals between them are calculated. The range of primary key values ​​located between two consecutive safe intervals, or before the first safe interval and after the last safe interval, constitutes the data gap. These gaps represent all data that has not been migrated or whose migration status is unknown.

[0056] The system service nodes acquire the real-time processing capacity status of all available migration service nodes. This status may include the node's remaining CPU / memory resources, network bandwidth, current number of tasks, and historical throughput for similar tasks. Simultaneously, the system service nodes analyze the inherent data characteristics of each data gap. For example, they estimate the number of rows within the gap by querying statistics from the source database and analyze the distribution density of primary key values. Then, the system service nodes execute matching and decision algorithms: for simple gaps with small data volume and low dispersion, they can be directly treated as a single shard; for complex gaps with large data volume or high dispersion, they are split into multiple sub-shards with balanced estimated processing times based on the processing capacity of available nodes; for multiple small and adjacent gaps, they can be merged into a single shard to improve the execution granularity of individual tasks and reduce task scheduling overhead.

[0057] After determining the primary key range of each data shard to be migrated, the system service node performs a boundary alignment optimization operation. First, it checks the start and end boundaries of each shard and fine-tunes these boundary values ​​to align them as precisely as possible with the boundaries of adjacent safe intervals. This ensures a clear separation between the sharding task and the already safe data, preventing any risk of duplicate or missed processing. Second, the system service node queries the physical storage metadata of the target database and further fine-tunes the shard boundaries to align them with the boundaries of the physical storage units of the target database. This embodiment of the application optimizes the I / O efficiency of subsequent migration service nodes performing batch writes to the target database through alignment optimization, reducing the writing of some pages and improving write performance and stability.

[0058] After boundary alignment is completed, the system service nodes generate a migration task description for each finalized, clearly defined data shard. This description includes the source table identifier, the precise primary key range, and the necessary transformation logic. Subsequently, the system service nodes allocate these tasks to available migration service nodes according to their scheduling strategy (such as weighted round-robin). During distribution, the shard's boundary information and the context of adjacent safe zones can be included to allow the migration service nodes to perform more refined verification or optimization during execution.

[0059] In one implementation of this application, the migration process is divided into multiple migration stages with dependencies. Based on the metadata of the objects to be migrated, a graph model reflecting the dependencies between tasks is constructed. The system service node updates the graph model according to the persistent task completion status and dynamically determines the set of tasks to be scheduled corresponding to the current migration stage based on the updated graph model state. The system service node schedules tasks sequentially in the set of tasks to be scheduled according to the dependency order of the migration stages, and after all execution tasks in the current migration stage have completed migration, it proceeds to schedule the execution tasks of the next migration stage.

[0060] Specifically, during the migration planning phase, the system service node parses the metadata of the objects to be migrated obtained from the source database. This metadata describes the definitions of database objects, such as tables, views, indexes, stored procedures, and users, and their dependencies. Based on this metadata, the system service node constructs a phase and task dependency graph model. Specifically, it categorizes different types of object migration tasks into different logical migration phases, such as: Phase 1: Migrating basic users and schemas; Phase 2: Migrating table structures without foreign key dependencies; Phase 3: Migrating table data with foreign key dependencies, etc. Nodes in the graph represent a migration phase or a specific migration task within a phase, and directed edges represent dependencies. For example, the phase in which the index creation task takes place necessarily depends on the phase in which its base table creation task takes place.

[0061] When resuming from a breakpoint, the system service node loads the previously saved stage and task dependency graph model, along with the completion status of all tasks, from persistent storage. The system service node traverses the dependency graph, removing all task nodes marked as completed and their outgoing edges from the currently active graph model. Next, the system service node identifies task nodes in this subgraph whose prerequisites have all been satisfied; these nodes constitute the currently schedulable task set.

[0062] System service nodes schedule tasks according to the inter-stage dependency order defined by the graph model, identifying the highest priority stage with available schedulable tasks. For this stage, the system service node employs a batch scheduling strategy: it selects a batch of tasks belonging to this stage from the currently schedulable task set and allocates these tasks for execution based on the number and load capacity of available migration service nodes. The system service node continuously monitors the completion status of all tasks within this stage. Only when it is confirmed through polling or callback that all executing tasks within the currently active stage have successfully completed migration does the system service node perform a stage state switch. Specifically, in the dependency graph model, this stage is marked as complete, triggering the graph model update logic. After the update, new tasks in the next dependency stage will have an in-degree of zero, thus being added to the schedulable task set, and the system service node will then begin scheduling tasks for this new stage. This process repeats until all stages are completed, thereby achieving phased migration progress.

[0063] Step 104: Based on the task information, the migration service node checks whether it has completed any unfinished tasks. If it has completed them, it returns the execution result to the system service node. If it has not completed them, it executes the task and returns the result to the system service node.

[0064] Specifically, after receiving an incomplete execution task assigned by the system service node, the migration service node first extracts the unique identifier of the task. Then, the migration service node queries its local persistent storage based on this task ID to retrieve whether there is any historical execution task information associated with this task. If the query returns a valid record indicating that the task's final execution status was successful, the node does not need to reconnect to the source and target databases to perform the actual data migration operation. Instead, it directly retrieves the historical execution result saved from local storage and reports the task completion message. If no local record for the task is found, or the record status is failure, the migration service node determines that the task is incomplete. The node then initiates the task execution process. After the task is completed, regardless of success or failure, the migration service node persists the execution result locally and updates the task's execution information. Finally, the node immediately constructs a reporting message based on the execution result and sends it to the system service node.

[0065] Figure 2 This application provides a schematic diagram of the stages and task allocation process during system service migration, as illustrated in the embodiments of this application. Figure 2As shown, the system service executes migration tasks, initializes the migration context, then iterates through the migration stages, initializes the current stage, persists the current migration stage, iterates through the objects to be migrated, generates migration tasks, and allocates tasks according to available migration services, recording which migration service each task is assigned to in a temporary table. After task allocation, it asynchronously waits for the results, updates the report based on the returned results, and removes the corresponding record from the temporary table. Once all tasks in the current stage are completed, it executes the release operation for that stage. Then, it checks if there is a next stage; if so, it switches to the next stage and repeats the above process; otherwise, it completes all migrations.

[0066] Figure 3 This is a schematic diagram illustrating the process of executing a migration task using a migration service, as provided in an embodiment of this application. Figure 3 As shown, after initialization is complete, the system receives the migration task assigned by the system service and then determines whether the task is in an "unknown" state. If so, it queries the local persistent record of the task to check the execution result of the task, and then checks whether the result exists. If yes, the result of the task is returned; if no, the task is executed and the result of the task is returned. If the initial determination of whether the task is in an "unknown" state is no, the task is executed directly and the result of the task is returned.

[0067] Figure 4 This application provides a schematic diagram of a system service breakpoint resume process, as shown in the embodiments. Figure 4 As shown, after executing the migration task, the migration context is initialized. Then, the process switches to the stage before the abnormal interruption in the persistent content. The current stage is then initialized and persisted. Based on the persistent content, unassigned and assigned migration tasks with no results are retrieved. Unassigned tasks are randomly assigned based on available migration service nodes. Assigned tasks with no results are reassigned to the previously used migration service and marked as "unknown." After task assignment, the process asynchronously waits for the results. Based on the returned results, the report is updated and the corresponding record for the task is removed from the temporary table. Once all tasks in the current stage are completed, the release operation for that stage is executed. Then, it is determined whether there is a next stage. If yes, the process switches to the next stage and repeats the above process; otherwise, all migrations are completed.

[0068] Figure 5 This is a schematic diagram of a breakpoint resume device for distributed migration of heterogeneous databases, provided as an embodiment of this application. Figure 5As shown, the breakpoint resume device 200 for heterogeneous database distributed migration includes: at least one processor 201; and a memory 202 communicatively connected to the at least one processor 201. The memory 202 stores instructions executable by the at least one processor 201, which, when executed, enable the at least one processor 201 to: persist migration execution information to a system service node; and persist execution task information to a migration service node receiving the task. The migration execution information includes at least one of migration execution stage information and execution task information allocated to the migration service node. In response to an abnormal interruption of the system service node and / or the migration service node, after the abnormal interruption node recovers, the system service node, based on the persisted migration execution information, determines the completed and incomplete execution tasks. It initiates the breakpoint resume process, whereby the system service node reassigns the determined incomplete execution tasks to the migration service node. The migration service node, based on the execution task information, checks whether it has completed the incomplete execution task; if completed, it returns the execution result to the system service node; if incomplete, it executes the task and returns the execution result to the system service node.

[0069] This application provides a non-volatile computer storage medium storing computer-executable instructions. These instructions are configured to: persist migration execution information by a system service node; and persist execution task information by a migration service node receiving a task. The migration execution information includes at least one of migration execution stage information and execution task information allocated to the migration service node. In response to an abnormal interruption of the system service node and / or the migration service node, after the abnormally interrupted node recovers, the system service node determines completed and incomplete execution tasks based on the persisted migration execution information. A breakpoint resume process is initiated, and the system service node reallocates the determined incomplete execution tasks to the migration service node. Based on the execution task information, the migration service node checks whether it has completed any incomplete execution tasks. If completed, it returns the execution result to the system service node; if incomplete, it executes the task and returns the execution result to the system service node.

[0070] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0071] The above descriptions are merely embodiments of this application and are not intended to limit the scope of this application. For those skilled in the art, various modifications and variations can be made to the embodiments of this application. These modifications or substitutions do not cause the essence of the corresponding technical solutions to depart from the spirit and scope of the technical solutions in the embodiments of this application.

Claims

1. A method for resuming interrupted migrations in a distributed migration of heterogeneous databases, characterized in that, The method is applied to a distributed migration architecture including at least one system service node and at least one migration service node, and the method includes: The system service node persists the migration execution information, and the migration service node that receives the task persists the execution task information; wherein the migration execution information includes at least one of the following: migration execution stage information and the execution task information allocated to the migration service node; In response to abnormal interruptions of system service nodes and / or migration service nodes, after the abnormally interrupted nodes recover, the system service nodes determine the completed and incomplete execution tasks based on the persistent migration execution information. Initiating the breakpoint resume process, the system service node will reallocate the identified unfinished execution tasks to the migration service node; Based on the execution task information, the migration service node checks whether it has completed the incomplete execution task. If it has completed the task, it returns the execution result to the system service node. If it has not completed the task, it executes the task and returns the execution result to the system service node. The system service node will reassign the identified unfinished execution tasks to the migration service node, specifically including: The system service node obtains a list of migration service nodes to be assigned based on the heartbeat information with each migration service node; The system service node assigns the unfinished execution tasks to nodes in the list of migration service nodes to be assigned, according to a pre-set weighted round-robin algorithm. In addition, when assigning tasks, the weight tokens of each node are released or distributed based on the number of tasks assigned to each node. Before assigning the incomplete execution task to a node in the list of migration service nodes to be assigned, the method further includes: The system service node initiates a global consistency snapshot query request to the target database to obtain the target end resource status view corresponding to the target database; wherein, the target end resource status view consists of the structure definition, data distribution overview and list of incomplete transaction IDs of the objects related to the current migration in the target database; The system service node analyzes the unfinished tasks and identifies conflicting task groups; wherein, the conflicting task groups include at least one of a data conflict group and a structural conflict group. For each of the unfinished execution tasks, the system service node simulates the changes to the target database status after executing the unfinished execution tasks based on the target resource status view, and generates a change version number based on the changes. Based on the changes in the target database status and the conflicting task groups, the system service node generates a conflict-free scheduling sequence corresponding to the incomplete execution task, and then assigns the conflict-free scheduling sequence to the corresponding node in the list of migration service nodes to be assigned.

2. The method for resuming interrupted migrations in a distributed migration of heterogeneous databases according to claim 1, characterized in that, In response to an abnormal interruption of the system service node and / or migration service node, after the abnormally interrupted node recovers, the system service node determines the completed and incomplete execution tasks based on the persistent migration execution information, specifically including: When the system service node is abnormally interrupted, and the migration service node is running normally or abnormally interrupted, the migration service node continues to execute tasks and saves results. After the system service node restarts, it resumes the execution phase before the abnormal interruption, and based on the migration execution information, it obtains the unassigned tasks and the assigned but not reported completed tasks in this phase, and reallocates the obtained tasks. When at least one of the migration service nodes is abnormally interrupted while the system service node is operating normally, the system service node stops assigning new tasks to the interrupted migration service node and marks the tasks already assigned to the interrupted migration service node as timeout unknown. When the interrupted migration service node comes back online, the system service node queries the relevant task results from the re-online node and updates the task status.

3. The method for resuming interrupted migrations in a distributed migration of heterogeneous databases according to claim 1, characterized in that, After assigning the conflict-free scheduling sequence to the corresponding node in the list of migration service nodes to be assigned, the method further includes: The system service node sends the version change number related to the incomplete execution task to the corresponding node in the list of migration service nodes to be assigned. Before executing the task, the migration service node determines the snapshot version information used by the system service node for inference based on the changed version number, and executes the task and reports the task success to the system service node when the current state of the relevant object in the target database is consistent with the snapshot version information. After receiving a successful task report from any migration service node, the system service node updates the global target resource status view, increments the relevant global version number, and broadcasts the global version number as an event to all online migration service nodes.

4. The method for resuming interrupted migrations in a heterogeneous database distributed migration according to claim 1, characterized in that, After the system service node determines the completed and incomplete execution tasks based on the persistent migration execution information, the method further includes: If the system service node detects an abnormal interruption and any shard migration task belonging to the same table has not been completed, the system service node will re-shard all data in that table. Based on the results of the resharding, new shard migration tasks are generated and assigned.

5. The method for resuming interrupted migrations in a heterogeneous database distributed migration according to claim 4, characterized in that, The system service node performs re-sharding on all data in the table, specifically including: The system service node determines the range of data that has been successfully migrated to the target table before the abnormal interruption and marks the verified continuous data range as a safe range; Based on the security interval, the system service node determines the data gaps in the entire table that have not been migrated and whose status is unknown; Based on the processing capacity status of the migration service node to be assigned the task and the data characteristics corresponding to the data gap, the system service node dynamically generates corresponding data shards to be migrated for each data gap; wherein, the data characteristics include at least one of primary key dispersion and data volume; The system service node performs boundary alignment operations on the data fragments to be migrated, so that the boundaries of the data fragments to be migrated are aligned with both the boundaries of the adjacent safe intervals and the boundaries of the physical storage units of the target database.

6. The method for resuming interrupted migrations in a heterogeneous database distributed migration according to claim 1, characterized in that, After the system service node reassigns the identified unfinished execution tasks to the migration service node, the method further includes: The migration process is divided into multiple dependent migration stages; Based on the metadata of the objects to be migrated, construct a graph model that reflects the dependencies between tasks; The system service node updates the graph model based on the persistent task completion status, and dynamically determines the set of tasks to be scheduled corresponding to the current migration stage based on the updated graph model status. The system service node schedules tasks sequentially from the set of tasks to be scheduled according to the dependency order of the migration phase, and after all the execution tasks of the current migration phase have completed the migration, it proceeds to schedule the execution tasks of the next migration phase.

7. A breakpoint resume device for distributed migration of heterogeneous databases, characterized in that, The device includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to perform the method described in any one of claims 1-6.

8. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are capable of performing the method described in any one of claims 1-6.

Images