Data migration method, system and equipment based on task arrangement

Abstract

The invention belongs to the technical field of computers, particularly relates to a data migration method, system and equipment based on task arrangement, and aims to solve the problems of difficulty in dynamic adjustment and lack of decoupling in the prior art. Data migration logic is packaged into a pluggable module, and a user arranges a migration process through a graphical interface: a module is selected, an execution sequence and dependency are set, and a structured task description file is generated. And the system automatically analyzes the dependency relationship in the file to generate an execution sequence. The parallel scheduling is independent of tasks; and after any task is completed, the postposition dependence is relieved, and the task meeting the condition is executed immediately. Abnormal response is carried out according to a strategy in the whole execution process, transaction rollback based on a persistent framework is automatically triggered when data operation is abnormal, and reliability is ensured. The method has the advantages of strong decoupling line and high expansibility, and the adaptability and maintainability of the migration process are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer, and particularly relates to a data migration method, system and device based on task arrangement. BACKGROUND

[0002] Data migration is of great significance in database upgrade, system migration, data integration and other scenarios. Existing data migration solutions usually include static script migration and data migration tools. Among them, the static script migration hard codes the data migration logic as SQL scripts or script files. This method is suitable for single migration task, but lacks flexibility and cannot respond to demand changes. Data migration tools provide certain migration process configuration functions. However, these tools are usually high in cost and difficult to meet the deep customization of specific needs. The common shortcomings are that it is difficult to dynamically adjust and lacks decoupling.

[0003] Based on this, the present application provides a data migration method, system and device based on task arrangement. SUMMARY

[0004] In order to solve the above problems in the prior art, that is, the problems of difficulty in dynamic adjustment and lack of decoupling in the prior art, the present application provides a data migration method, system and device based on task arrangement.

[0005] In the first aspect of the present application, a data migration method based on task arrangement is provided, which comprises:

[0006] Each data migration logic is encapsulated as an independent function module and deployed in the form of a plug-in file;

[0007] A migration task process is created through a task arrangement interface, the migration tasks to be executed are selected from the deployed function modules, the execution order of the function modules is set according to the business dependency relationship or user-defined operation, and a structured task description file containing task node attributes and dependency relationships is generated;

[0008] The dependency relationship in the structured task description file is parsed to generate a task sequence executed in the order of dependency;

[0009] The task sequence is traversed, all tasks without pre-dependencies are detected and executed; after the execution of any task is completed, the corresponding dependency markers in all other tasks with the task as pre-dependencies are removed, and whether these tasks have no pre-dependencies left is detected; if not, the task is executed; and tasks without dependency relationships can be executed in parallel;

[0010] During the execution, an exception response is performed according to the error handling strategy defined in the structured task description file, and a transaction rollback operation based on a persistent framework is triggered when a data operation exception occurs.

[0011] Further, the function module is packaged through a preset task implementation interface, in which a multi-data source connection configuration of a source database and a target database, a new-old table field mapping rule and a migration condition judgment logic are defined.

[0012] Further, the structured task description file comprises a plurality of task nodes, each node having a unique identifier field, a task type field, a parameter configuration field and a preposed dependent identifier set field.

[0013] Further, the error processing strategy comprises any one of the following modes:

[0014] Automatically retrying a specified number of times when failing, or ignoring errors and continuing to execute subsequent tasks, or terminating the entire task flow.

[0015] Further, the error processing strategy supports global configuration and task-level independent configuration, specifically:

[0016] The global strategy is applied to all tasks;

[0017] The task-level strategy covers the global strategy and defines an independent error processing mode for a specific task.

[0018] Further, the method further comprises a migration monitoring mechanism, and the migration monitoring mechanism comprises:

[0019] Task start time, task end time, description information and execution state are displayed through an independent monitoring interface;

[0020] The task state is updated in real time to be in execution, success or failure.

[0021] Further, the transaction rollback operation comprises:

[0022] Triggering transaction rollback when data query or insertion is abnormal;

[0023] Clearing the migrated abnormal data and recording error logs.

[0024] Further, the method supports an incremental migration mode, and the incremental migration mode comprises:

[0025] Marking an incremental migration task in the structured task description file;

[0026] Migrating only newly added or changed data according to the configuration parameters.

[0027] The second aspect of the application provides a data migration system based on task arrangement, based on a data migration method based on task arrangement, the system comprises:

[0028] The plug-in encapsulation module is configured to encapsulate each data migration logic as an independent functional module and deploy the functional module in the form of a plug-in file.

[0029] The task arrangement interface module is configured to create a migration task flow through a task arrangement interface and select a migration task to be executed from the deployed functional modules.

[0030] The description file generation unit is configured to set the execution order between the functional modules according to a business dependency relationship or a user-defined operation, and generate a structured task description file containing task node attributes and dependency relationships.

[0031] The dependency analysis unit is configured to analyze the dependency relationships in the structured task description file and generate a task sequence executed in a dependency order.

[0032] The task execution unit is configured to traverse the task sequence, detect and execute all tasks without a preceding dependency, remove the corresponding dependency markers in all other tasks with the task as a preceding dependency after the execution of any task is completed, and detect whether the tasks have no uncompleted preceding dependencies; if not, the task is executed; and the task execution unit also supports parallel execution of tasks without a dependency relationship.

[0033] The exception handling unit is configured to respond to an exception according to an error handling strategy defined in the structured task description file during execution, and trigger a transaction rollback operation based on a persistent framework when a data operation exception occurs.

[0034] In a third aspect, the application provides an electronic device, comprising:

[0035] at least one processor; and

[0036] a memory communicatively connected to the at least one processor; wherein

[0037] the memory stores instructions executable by the processor, and the instructions are used to implement a task arrangement-based data migration method when executed by the processor.

[0038] The application has the following advantages:

[0039] (1) High flexibility: the migration flow is dynamically configured through the structured task description file, the user can adjust the task dependency relationship, error handling strategy and incremental migration marker at any time according to the business requirements, the underlying code does not need to be modified to respond to frequent changes in migration requirements, and the adaptability and maintainability of the migration flow are significantly improved.

[0040] (2) Strong decoupling: Each data migration logic is encapsulated as an independent plug-in file. Modules interact through standardized interfaces. Modification, replacement, or expansion of any module does not affect the overall system operation, achieving physical isolation and functional autonomy of migration tasks, and significantly reducing system coupling.

[0041] (3) High scalability: It supports user-defined migration task interfaces. By developing functional modules that comply with specifications and deploying them as plug-in files, it can seamlessly integrate new migration logic into existing task sequences without the need to restructure the system architecture, effectively supporting the continuous expansion of heterogeneous database migration scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0043] Figure 1 This is a flow chart of a data migration method based on task scheduling of the present invention. DETAILED DESCRIPTION

[0044] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0045] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0046] A first embodiment of the present invention provides a data migration method based on task scheduling, the method comprising:

[0047] Encapsulate each data migration logic into an independent functional module and deploy it as a plug-in file;

[0048] Create a migration task process through the task orchestration interface. Select the migration tasks to be executed from the deployed functional modules. Set the execution order between functional modules based on business dependencies or user-defined operations. Generate a structured task description file containing task node attributes and dependencies.

[0049] Parsing the dependency relationship in the structured task description file to generate a task sequence to be executed in the dependency order;

[0050] Traverse the task sequence, detect and execute all tasks without pre-dependencies; after the completion of any task execution, remove all corresponding dependency markers in other tasks that have the task as a pre-dependency, and detect whether these tasks have no uncompleted pre-dependencies; if not, execute the task; meanwhile, support parallel execution of tasks without dependency relationships;

[0051] During execution, perform abnormal response according to the error handling strategy defined in the structured task description file, and trigger transaction rollback operation based on the persistent framework when a data operation is abnormal.

[0052] In order to more clearly describe the data migration method based on task arrangement, the following will be combined with Figure 1 The embodiment of the application is described in detail as follows:

[0053] Each data migration logic is encapsulated as an independent functional module and deployed in the form of a plug-in file.

[0054] The functional module is encapsulated through a preset task implementation interface, and in the preset task implementation interface, the multi-data source connection configuration of the source database and the target database, the new-old table field mapping rule, and the migration condition judgment logic are defined.

[0055] In the implementation process of the application, the first step is to encapsulate the code for implementing specific data migration logic into an independent and reusable functional module. Each such module focuses on completing a specific and fine-grained data migration operation, such as migrating data of a database table or executing a specific data conversion script.

[0056] The core of encapsulation is to implement a predefined task function interface using the Java programming language (or similar technologies that support interfaces and packaging). This interface requires each module to include a specific implementation of its migration logic, and requires the source database and target database connection information (usually implemented by configuring a multi-data source identifier), the target table name to be migrated, and related metadata information such as task name, description, etc.

[0057] In the implementation of specific migration logic, the developer needs to define data objects representing source table fields and target table fields respectively according to the new and old database table structures, and establish actual connections with the database through JDBC (Java Database Connectivity), complete the complete operation process of querying data from the source table, converting and filling the source table field values to the target table field objects according to the predefined mapping relationship, and finally inserting the data into the corresponding table in the target database.

[0058] After the code development and testing are completed, each independent migration function module is compiled and packaged into a standard JAR (Java Archive) file by using the package command of the Maven build tool (or a similar tool). The packaged JAR file is the "plug-in file" described in the present application, which contains complete migration logic implementation and necessary configuration information. Finally, the generated JAR plug-in file is uploaded and deployed to the designated repository of the task scheduling and management platform (or system) built in the present application, for subsequent selection and loading for execution during task scheduling.

[0059] A migration task flow is created through a task scheduling interface, migration tasks to be executed are selected from the deployed function modules, the execution order of the function modules is set according to business dependency relationships or user-defined operations, and a structured task description file containing task node attributes and dependency relationships is generated.

[0060] In this embodiment, after the plug-in deployment of the migration function modules is completed, a user creates a specific migration task flow through a graphical task scheduling interface provided by the system.

[0061] In this interface, the user first selects the specific migration tasks (i.e., small tasks such as migrating the user table, the role table, etc.) to be executed from the function module library (i.e., the collection of uploaded JAR plug-ins) deployed by the platform according to the actual needs of the current data migration. After selecting the tasks, the user needs to set the execution order and dependency relationships of the selected task nodes on the interface according to the actual business logic dependency relationships (for example, the migration of the user role association table must be performed after the migration of the user table and the role table is completed) or according to self-management needs for custom operations.

[0062] This setting is usually achieved by dragging the task nodes and drawing dependency lines between the nodes to explicitly specify that a certain task must be started after the successful completion of the preceding tasks. The system converts all the above scheduling operations of the user (including the selected task nodes, the attribute configuration of each node, and the dependency relationships between the nodes) into a structured machine-readable description file in real time.

[0063] The description file is organized in JSON format, which clearly defines the topology of the entire task flow. The JSON file contains an array of objects, each object containing the following key attributes: id (unique identifier of the task), name (task name for interface display), type (task type such as "SQL query", "script execution" or specific plugin type identifier), config (an object containing all specific configuration parameters required for task execution, such as source / target database connection information, table name, migration conditions, etc.), dependencies (an array listing all pre-task ids that the task depends on, indicating strict execution order constraints). Finally, this generated JSON file completely describes the execution blueprint of the entire migration task, serving as the basis for subsequent task scheduling engine parsing and execution.

[0064] Parsing the dependencies in the structured task description file to generate a task sequence for execution in the order of dependencies;

[0065] In this embodiment, the process of parsing the dependencies in the structured task description file and generating a task sequence includes:

[0066] Load and parse the structured task description file, identify all task nodes defined in the file and their included pre-dependency definition items, which are lists containing the unique identifiers of the pre-tasks that the task depends on;

[0067] Based on the parsed task nodes and pre-dependency definition items (dependencies attribute), dynamically build a directed acyclic graph (DAG) in memory to represent the dependencies between tasks, where task nodes are vertices of the graph, and pre-tasks that the task node depends on (indicated by pre-dependency definition items) are edges pointing to the task node;

[0068] Traverse the directed acyclic graph and continuously detect task nodes with an in-degree of zero based on the in-degree-based topological sorting algorithm, where an in-degree of zero indicates that the task node has no outstanding pre-tasks or its pre-dependency definition items are empty;

[0069] Add the detected tasks with an in-degree of zero to the task queue in order to generate a task sequence that strictly follows the dependencies, ensuring that tasks with dependencies are executed in the defined order, and marking tasks without dependencies as a parallel execution candidate set.

[0070] Specifically, after generating the structured task description file (JSON format) that describes the entire migration task flow, the task orchestration engine (or core scheduling module) will load and parse the file.

[0071] The parsing process first identifies all the task nodes defined in the file and their attributes (such as id, name, type, config). The most critical is to parse the dependencies attribute (an array containing the id of the pre-task) defined in each task node, which explicitly indicates all the pre-tasks that must be successfully completed before the task starts execution.

[0072] Based on the parsed task node information and their dependencies relationship, the engine dynamically builds a directed acyclic graph (DAG) in memory that reflects the dependencies between tasks. In the graph, each task node is a vertex, and the dependencies define the directed relationship that forms the edge (for example, task B's dependencies contains the id of task A, which means there is an edge from A to B, meaning A must be completed before B).

[0073] The engine then traverses the entire DAG and determines the reasonable order of execution of all tasks through a specific algorithm (such as a topological sorting algorithm based on in-degree). The core logic of this algorithm is: continuously detect and find all "in-degree zero" task nodes (i.e. tasks that have no unfinished pre-tasks or pre-task list dependencies is empty), identify these tasks that have no dependencies or all dependencies have been met. The engine adds these immediately executable tasks to a sequence (or queue) of tasks to be executed in the order detected (or added).

[0074] This generated sequence strictly follows the dependencies, ensuring that tasks with dependencies are executed in the order defined, while those tasks that have no dependencies between each other (tasks with the same in-degree) are identified as candidates for parallel execution.

[0075] The final generated task sequence organized in the order of dependencies provides a clear instruction set for the next step of task execution scheduling. The engine will strictly follow the instructions of this sequence in the subsequent execution phase, prioritizing the execution of tasks in the sequence that have no pre-dependencies or dependencies that have been met.

[0076] Traverse the task sequence, detect and execute all tasks with no pre-dependencies; after the completion of any task execution, remove the corresponding dependency flag in all other tasks that have the task as a pre-dependency, and detect whether these tasks have no unfinished pre-dependencies; if not, execute the task; while supporting parallel execution of tasks with no dependencies;

[0077] In this embodiment, traversing the task sequence specifically includes:

[0078] The task scheduling engine screens the tasks that can be executed at present according to the task sequence, and the executable tasks need to meet the requirement of no pre-dependence or all pre-tasks have been successfully executed;

[0079] The screened executable tasks are scheduled and allocated:

[0080] For the tasks without dependence, the thread pool concurrent mechanism is used to start the execution at the same time;

[0081] For the tasks with dependence, the execution is triggered after the pre-task state is confirmed to be successfully completed in strict sequence order;

[0082] The JAR plug-in file corresponding to each task is dynamically loaded, and the predefined task function interface is called to execute the migration logic;

[0083] The task execution state is recorded and updated in real time to the shared state manager;

[0084] The process of "detecting executable tasks, scheduling and executing, updating states, and triggering subsequent tasks" is continuously executed through a loop detection mechanism until the task sequence is completely executed.

[0085] Specifically, the task scheduling engine starts to execute the migration tasks according to the generated task sequence. The engine first screens all the tasks that can be executed at present from the task queue, and the characteristics of these tasks are that there is no pre-dependence requirement (i.e. the nodes with in-degree of zero in the topological sorting), or all the pre-tasks on which they depend have been successfully executed. For the screened executable tasks, the engine schedules and allocates them according to the system resource status (such as the number of available threads, load condition).

[0086] Among them, the tasks without dependence are identified as parallel execution units, and the engine starts the execution of these tasks at the same time through the thread pool and other concurrent mechanisms to maximize the use of computing resources and improve the overall migration efficiency. For the tasks with dependence, the engine strictly follows the order specified in the task sequence, and only after the pre-task state is confirmed to be "successfully completed", the subsequent dependent tasks are added to the executable queue.

[0087] The method further includes a migration monitoring mechanism, and the migration monitoring mechanism includes:

[0088] The task start time, task end time, description information, and execution state are displayed through an independent monitoring interface;

[0089] The task state is updated in real time to be in execution, success, or failure.

[0090] The execution process of each task is essentially to dynamically load its corresponding JAR plug-in file and call the predefined task function interface implementation therein. The task execution result state (success, failure, execution in progress) is recorded and updated to the shared state manager in real time. The engine continuously monitors the task queue and the task execution state, and repeatedly executes the process of "detecting executable tasks -> scheduling execution (serial or parallel) -> updating the state -> triggering the subsequent tasks" until all tasks in the task sequence are executed.

[0091] During execution, the error handling strategy defined in the structured task description file is used for abnormal response, and the transaction rollback operation based on the persistent framework is triggered when a data operation exception occurs.

[0092] In this embodiment, the error handling strategy includes any of the following modes:

[0093] Automatically retry a specified number of times when failing; or ignore the error and continue to execute the subsequent task; or terminate the entire task process.

[0094] Specifically, the task scheduling engine monitors the execution state of each task in real time during the execution of the task sequence. When an abnormal task execution is detected, the engine will respond according to the error handling strategy predefined in the structured task description file. The strategy supports three processing modes:

[0095] Failure retry: automatically retry a specified number of times (such as configuring 3 retries) when the task execution fails, until success or the maximum number of retries is reached;

[0096] Ignore error: skip the current failed task and continue to execute the subsequent independent task;

[0097] Process termination: immediately terminate the entire migration task process and mark it as a failure state.

[0098] In this embodiment, the error handling strategy supports global configuration and task-level independent configuration. Specifically:

[0099] Global strategy is applied to all tasks;

[0100] Task-level strategy overrides the global strategy and defines an independent error handling mode for a specific task.

[0101] Specifically, the error handling strategy uses a hierarchical configuration mechanism:

[0102] Global strategy: define the default handling rule at the root node of the task description file, which is applicable to all tasks that are not configured separately;

[0103] Task-level policy: Define the processing method independently in the configuration attribute of a single task node (for example, set "retry": 3 in the errorPolicy field of the JSON node), and this policy will override the global policy.

[0104] In this embodiment, the transaction rollback operation includes:

[0105] Trigger transaction rollback when data query or insertion exception occurs;

[0106] Clear the migrated exception data and record error logs.

[0107] Specifically, for the migration task involving database operation (such as SQL query or data insertion), if data operation exception (such as primary key conflict, data type error) occurs during execution, the engine will trigger the transaction rollback operation based on the persistent framework. Specifically, it includes:

[0108] Roll back the database transaction unit where the current task is located through the Spring transaction manager (or similar framework);

[0109] Automatically clear part of the written data in the target database due to exception, ensuring data consistency;

[0110] Record detailed error logs to system log files and database error tables, including exception stack, failed data batch, associated task ID, and other key information.

[0111] The engine continuously applies the above strategy in the execution process until all tasks are successfully completed or the process is terminated according to the strategy.

[0112] Although the above steps are described in the above order in the above embodiment, those skilled in the art can understand that, in order to achieve the effect of the embodiment, the different steps do not have to be executed in such an order, they can be executed simultaneously (in parallel) or in a reversed order, and these simple changes are within the protection scope of the present application.

[0113] The second embodiment of the present application is a data migration system based on task arrangement, based on the first embodiment of the data migration method based on task arrangement, the system includes:

[0114] The plug-in encapsulation module is configured to encapsulate each data migration logic as an independent functional module and deploy it in the form of a plug-in file;

[0115] The task arrangement interface module is configured to create a migration task flow through the task arrangement interface and select the migration tasks to be executed from the deployed functional modules;

[0116] The description file generation unit is configured to set the execution order between the functional modules according to the business dependency relationship or user self-defined operation, and generate a structured task description file containing task node attributes and dependency relationships;

[0117] The dependency analysis unit is configured to analyze the dependency relationship in the structured task description file, and generate a task sequence executed in the dependency order.

[0118] The task execution unit is configured to traverse the task sequence, detect and execute all tasks without pre-dependencies; after the execution of any task is completed, the corresponding dependency marks in all other tasks with the task as a pre-dependency are removed, and it is detected whether these tasks have no pre-dependency to be completed; if not, the task is executed; and the tasks without dependency relationships are supported to be executed in parallel.

[0119] The exception handling unit is configured to perform abnormal response according to the error handling strategy defined in the structured task description file during execution, and trigger transaction rollback operation based on the persistent framework when data operation is abnormal. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related description of the above-described system can be referred to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0120] It should be noted that the above embodiment provides a task arrangement-based data migration system, which is only exemplified by the division of the above functional modules. In actual application, the above functions can be completed by different functional modules according to needs, that is, the modules or steps in the embodiment of the present application are further divided or combined, for example, the modules of the above embodiment can be combined into one module, or can be further split into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present application are only for distinguishing the modules and steps, and should not be considered as an improper limitation of the present application.

[0121] The third embodiment of the electronic device comprises:

[0122] at least one processor; and

[0123] a memory in communication connection with the at least one processor; wherein

[0124] The memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned task arrangement-based data migration method.

[0125] The fourth embodiment of the application is a computer readable storage medium storing computer instructions for being executed by a computer to implement the task orchestration based data migration method.

[0126] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the storage device and the processing device described above and the related descriptions can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0127] Those skilled in the art should appreciate that the modules and method steps of each example described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. The software modules and method steps corresponding to the various examples can be stored in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of each example have been described in general terms above. Whether the functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0128] The terms "first", "second", and the like are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0129] The term "comprising" or any other similar term is intended to encompass non-exclusive inclusion, so that a process, method, article, or device / apparatus including a series of elements includes not only those elements, but also other elements not explicitly listed, or inherent to the process, method, article, or device / apparatus.

[0130] So far, the technical solution of the application has been described in combination with the preferred embodiments shown in the drawings, but those skilled in the art can easily understand that the protection scope of the application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to the related technical features without departing from the principles of the application, and the technical solutions after the changes or replacements will fall within the protection scope of the application.

Claims

1. A data migration method based on task scheduling, characterized in that: The method includes: Encapsulate each data migration logic into an independent functional module and deploy it as a plug-in file; Create a migration task process through the task orchestration interface. Select the migration tasks to be executed from the deployed functional modules. Set the execution order between functional modules based on business dependencies or user-defined operations. Generate a structured task description file containing task node attributes and dependencies. Parsing the dependency relationship in the structured task description file to generate a task sequence to be executed in the dependency order; Traverse the task sequence, detect and execute all tasks without predecessor dependencies; after any task is completed, remove the corresponding dependency marks of all other tasks that have this task as a predecessor dependency, and detect whether these tasks have no unfinished predecessor dependencies; if not, execute the task; also supports parallel execution of tasks without dependencies; During the execution process, the system responds to exceptions according to the error handling strategy defined in the structured task description file, and triggers a transaction rollback operation based on the persistence framework when data operations are abnormal.

2. The data migration method based on task scheduling according to claim 1, characterized in that: The functional module is encapsulated through a preset task implementation interface, in which the multi-data source connection configuration of the source database and the target database, the new and old table field mapping rules and the migration condition judgment logic are defined.

3. The data migration method based on task scheduling according to claim 1, characterized in that: The structured task description file includes a plurality of task nodes, each of which has a unique identifier field, a task type field, a parameter configuration field, and a pre-dependency identifier set field.

4. The data migration method based on task scheduling according to claim 1, characterized in that: The error handling strategy includes any of the following methods: Automatically retry a specified number of times upon failure; or ignore the error and continue executing subsequent tasks; or terminate the entire task process.

5. The data migration method based on task scheduling according to claim 1, characterized in that: The error handling strategy supports global configuration and task-level independent configuration. Specifically: The global policy applies to all tasks; Task-level policies override the global policy and define independent error handling for a specific task.

6. The data migration method based on task scheduling according to claim 1, characterized in that: The method further includes a migration monitoring mechanism, wherein the migration monitoring mechanism includes: The task start time, task end time, description information and execution status are displayed through an independent monitoring interface; The task status is updated in real time as in progress, successful, or failed.

7. The data migration method based on task scheduling according to claim 1, characterized in that: The transaction rollback operation includes: Trigger transaction rollback when data query or insertion is abnormal; Clear the migrated abnormal data and record the error log.

8. The data migration method based on task scheduling according to claim 1, characterized in that: The method supports an incremental migration mode, which includes: Mark incremental migration tasks in a structured task description file; Migrate only new or changed data based on configuration parameters.

9. A data migration system based on task scheduling, based on the data migration method based on task scheduling according to any one of claims 1 to 8, characterized in that: The system includes: A plug-in encapsulation module is configured to encapsulate each data migration logic into an independent functional module and deploy it in the form of a plug-in file; A task scheduling interface module is configured to create a migration task process through the task scheduling interface and select the migration task to be executed from the deployed functional modules; A description file generating unit configured to set the execution order between functional modules according to business dependencies or user-defined operations, and generate a structured task description file containing task node attributes and dependencies; A dependency parsing unit configured to parse the dependency relationship in the structured task description file and generate a task sequence to be executed in the dependency order; The task execution unit is configured to traverse the task sequence, detect and execute all tasks without predecessor dependencies; after any task is completed, it removes the corresponding dependency markers of all other tasks that have this task as a predecessor dependency, and detects whether these tasks have no unfinished predecessor dependencies; if not, it executes the task; it also supports the parallel execution of tasks without dependencies; The exception handling unit is configured to respond to exceptions according to the error handling strategy defined in the structured task description file during execution, and trigger a transaction rollback operation based on the persistence framework when data operations are abnormal.

10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the data migration method based on task scheduling as described in any one of claims 1-8.

Images