A real-time data synchronization task scheduling optimization method and system

Abstract

The application discloses a real-time data synchronization task scheduling optimization method and system, and relates to the technical field of big data processing. In the method, first, a first execution priority sequence and a previous state of a synchronization task are generated according to a first foreign key dependency feature of a source database table; second, a task distribution plan is generated and executed in combination of resource requirements of the task and available resources of a target node; in the execution process, if the load of the target node exceeds the limit, the unexecuted task is dynamically migrated to a backup node; then, operation change logs generated by each node are acquired, and the execution time sequence and the priority sequence are compared, if consistent, the synchronization is confirmed to be successful and the source database is updated; finally, when the foreign key dependency feature changes, a second execution priority sequence and a previous state can be dynamically updated. The technical scheme provided by the application can improve the accuracy of data synchronization task scheduling in processing complex business scenarios.

Description

Technical Field

[0001] This invention belongs to the field of big data processing technology, specifically a method and system for optimizing real-time data synchronization task scheduling. Background Technology

[0002] With the deepening of enterprise digital transformation and the rapid development of cloud computing technology, data has become a core element driving business innovation. Especially in the era of big data, the massive amounts of data generated by enterprises daily are growing exponentially. Efficiently processing and synchronizing petabyte-scale (PB) or even exabyte-scale (EB) data volumes has become a key technological challenge. In practical applications, enterprises often need to perform real-time data synchronization between heterogeneous database systems to support various business scenarios such as multi-site deployment, disaster recovery, and data analysis. Among these, parallel synchronization technology based on task scheduling has been widely used in scenarios requiring large-scale real-time data synchronization, such as e-commerce platforms and financial systems, due to its high execution efficiency and full resource utilization.

[0003] Currently, mainstream data synchronization systems primarily employ a task distribution mechanism based on resource matching. This scheduling method allocates synchronization tasks according to the available resources of each execution node at system startup. Execution begins as soon as node resources meet the task requirements, shortening the overall synchronization time through parallel processing by multiple nodes and providing enterprises with efficient data transmission capabilities.

[0004] However, foreign key constraints are commonly present between tables in relational databases. In actual synchronization processes, due to significant differences in database architecture and table dependencies among different enterprises, existing task scheduling mechanisms struggle to guarantee that the execution order of synchronization tasks meets data integrity requirements. This leads to frequent constraint violations during parallel execution across multiple nodes, causing synchronization failures and data inconsistencies, thus reducing the accuracy of data synchronization task scheduling in handling complex business scenarios. Summary of the Invention

[0005] To address the above problems, this invention provides a real-time data synchronization task scheduling optimization method and system, which solves the problem that the task scheduling mechanism in the prior art cannot guarantee that the execution order of synchronization tasks meets the data integrity requirements, resulting in frequent constraint violation errors. This invention can improve the accuracy of data synchronization task scheduling when handling complex business scenarios.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: Obtain the first foreign key dependency features of each data table in the source database, generate the first execution priority sequence of the synchronization task based on the first foreign key dependency features, and determine the first pre-execution state of the synchronization task. Extract the resource requirement identifier of the synchronization task, obtain the current resource quantity of the target node, and distribute the synchronization task to the target node according to the resource requirement identifier and the current resource quantity to obtain a task distribution plan; The synchronization task is executed on the target node according to the task distribution plan. When the operating load of the target node is greater than the preset load threshold, the unexecuted synchronization tasks on the target node are migrated to the backup node, and the task receiving status of the backup node is recorded. Obtain the operation change logs generated by the backup node and the target node during the execution of the synchronization task, and compare the execution time sequence of the operation change logs with the sorting result of the first execution priority sequence; When the execution time sequence is the same as the sorting result, a synchronization success confirmation signal is generated, and the synchronization completion flag of the source database is updated according to the synchronization success confirmation signal; When the first foreign key dependency feature changes, a second execution priority sequence is generated based on the changed second foreign key dependency feature, and a second preceding execution state for each synchronous task is determined.

[0007] By adopting the above technical solution, before executing the data synchronization task, the first foreign key dependency characteristics of each data table in the source database are obtained. Based on these characteristics, a first execution priority sequence and a first pre-execution state are generated, enabling the system to identify the dependencies between data tables and establish clear execution order constraints for the synchronization task. This ensures that the data in the dependent master table is synchronized first, thus avoiding foreign key constraint violations caused by improper execution order. Furthermore, by extracting the resource requirement identifier of the synchronization task and combining it with the current resource availability of the target node, task distribution is performed, ensuring that the task execution order conforms to the dependency requirements and achieving reasonable resource allocation. When the operating load of the target node exceeds a preset load threshold, unexecuted synchronization tasks are promptly migrated to a backup node, and the task reception status is recorded, effectively addressing the impact of node load fluctuations on synchronization stability. Further, by comparing the execution time sequence of the operation change log with the sorting result of the first execution priority sequence, and generating a synchronization success confirmation signal to update the synchronization completion flag in the source database when they are identical, a dual verification of the task execution correctness is achieved, ensuring data integrity and consistency. Meanwhile, when the first foreign key dependency feature changes, the system can regenerate the second execution priority sequence and determine the second pre-execution state based on the changed second foreign key dependency feature, so that the scheduling strategy can dynamically adapt to changes in the database structure, thereby improving the accuracy of data synchronization task scheduling when dealing with complex business scenarios.

[0008] Optionally, the primary table identifier and secondary table identifier are extracted from the first foreign key dependency features; the synchronization task corresponding to the primary table identifier is determined as the pre-execution item, and the synchronization task corresponding to the secondary table identifier is determined as the post-execution item; the execution order of the pre-execution items and the post-execution items is arranged to obtain the first execution priority sequence; in the first execution priority sequence, the synchronization task corresponding to the secondary table identifier is obtained, and the target synchronization task corresponding to each primary table identifier associated with the synchronization task corresponding to the secondary table identifier is found; the execution completion status of each target synchronization task corresponding to the primary table identifier is extracted, and the execution completion status is determined as the first pre-execution status of the synchronization task corresponding to the secondary table identifier.

[0009] Optionally, the estimated data processing volume of the synchronization task is obtained, and the estimated data processing volume is used as the resource requirement identifier; the remaining memory and processor idle rate of multiple candidate nodes are obtained, and the current resource quantity is determined based on the remaining memory and processor idle rate; candidate nodes whose resource requirement identifier is less than the current resource quantity are selected as target nodes; and the task distribution plan is generated according to the matching result between the synchronization task and the target node.

[0010] Optionally, when the operating load of the target node exceeds a preset load threshold, a task pause command is sent to the target node; the task execution progress identifier of the target node when it receives the task pause command is obtained; unexecuted synchronous tasks are extracted based on the task execution progress identifier, and the unexecuted synchronous tasks are sent to the backup node; when the backup node returns a successful reception response, the successful reception response is recorded as the task reception status.

[0011] Optionally, the standby node identifier in the task receiving status is extracted; the node routing information in the task distribution plan is updated according to the standby node identifier; and a task recovery instruction is sent to the standby node so that the standby node continues to execute the unexecuted synchronization task based on the updated node routing information.

[0012] Optionally, extract the data table identifiers and operation completion timestamps from the operation change log; arrange the data table identifiers according to the operation completion timestamps to obtain the execution time sequence of the operation change log; extract the corresponding data table identifiers sequentially according to the execution priority of each synchronous task in the first execution priority sequence to generate the sorting result; compare the execution time sequence of the operation change log with the data table identifiers of the same position in the sorting result item by item to obtain the comparison result.

[0013] Optionally, when the execution time sequence is the same as the sorting result, the transaction commit identifier in the operation change log is extracted; the transaction commit identifier is bound to the table name of the data table to generate the synchronization success confirmation signal; the synchronization success confirmation signal is sent to the control terminal of the source database so that the control terminal modifies the status field corresponding to the data table to the synchronization completion flag.

[0014] Optionally, the table structure definition file of the source database is read; the current foreign key dependency feature in the table structure definition file is extracted; the current foreign key dependency feature is compared with the first foreign key dependency feature; when the current foreign key dependency feature is different from the first foreign key dependency feature, the current foreign key dependency feature is used as the second foreign key dependency feature, and the second execution priority sequence is generated according to the second foreign key dependency feature.

[0015] Optionally, a metadata read request is sent to the source database, and a metadata set returned by the source database is received; constraint definition fields of each data table are extracted from the metadata set; for each data table, the association field corresponding to the primary key of the associated data table in the source database is extracted from the constraint definition fields, and the association field is used as the first foreign key dependency feature of the data table.

[0016] In a second aspect, embodiments of this application provide a real-time data synchronization task scheduling optimization system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, which includes computer instructions, and the one or more processors call the computer instructions to cause the real-time data synchronization task scheduling optimization system to perform the method described in the first aspect and any possible implementation thereof.

[0017] In summary, one or more technical solutions provided in this application have at least the following technical effects or advantages: By adopting the above technical solution, before executing the data synchronization task, the first foreign key dependency characteristics of each data table in the source database are obtained. Based on these characteristics, a first execution priority sequence and a first pre-execution state are generated, enabling the system to identify the dependencies between data tables and establish clear execution order constraints for the synchronization task. This ensures that the data in the dependent master table is synchronized first, thus avoiding foreign key constraint violations caused by improper execution order. Furthermore, by extracting the resource requirement identifier of the synchronization task and combining it with the current resource availability of the target node, task distribution is performed, ensuring that the task execution order conforms to the dependency requirements and achieving reasonable resource allocation. When the operating load of the target node exceeds a preset load threshold, unexecuted synchronization tasks are promptly migrated to a backup node, and the task reception status is recorded, effectively addressing the impact of node load fluctuations on synchronization stability. Further, by comparing the execution time sequence of the operation change log with the sorting result of the first execution priority sequence, and generating a synchronization success confirmation signal to update the synchronization completion flag in the source database when they are identical, a dual verification of the task execution correctness is achieved, ensuring data integrity and consistency. Meanwhile, when the first foreign key dependency feature changes, the system can regenerate the second execution priority sequence and determine the second pre-execution state based on the changed second foreign key dependency feature, so that the scheduling strategy can dynamically adapt to changes in the database structure, thereby improving the accuracy of data synchronization task scheduling when dealing with complex business scenarios. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a real-time data synchronization task scheduling optimization method disclosed in an embodiment of this application; Figure 2 This is another flowchart illustrating a real-time data synchronization task scheduling optimization method disclosed in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a system provided in an embodiment of this application.

[0019] In the diagram: 301, Central Processing Unit; 302, Read-Only Memory; 303, Random Access Memory; 304, Bus; 305, Input / Output Interface; 306, Input Section; 307, Output Section; 308, Storage Section; 309, Communication Section; 310, Driver; 311, Removable Media. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solution, the present invention will be described in detail below with reference to embodiments. The description in this part is only exemplary and explanatory, and should not be used to limit the scope of protection of the present invention in any way.

[0021] It should be noted that, in this document, the terms "comprising," "including," and any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Specific examples have been used in this document to illustrate the principles and implementation methods of the present invention. These examples are merely for the purpose of helping to understand the method and core ideas of the present invention. The above are only preferred embodiments of the present invention. It should be pointed out that, due to the limitations of written expression and the objective existence of infinite specific structures, those skilled in the art can make several improvements, modifications, or variations without departing from the principles of the present invention, and can also combine the above technical features in an appropriate manner. These improvements, modifications, variations, or combinations, or the direct application of the concept and technical solution of the present invention to other situations without modification, should all be considered within the scope of protection of the present invention.

[0022] This application provides a method for optimizing the scheduling of real-time data synchronization tasks, referring to... Figure 1 , Figure 1 This is a flowchart illustrating a real-time data synchronization task scheduling optimization method provided in an embodiment of this application. The method is applied to a system, which refers to a hardware and software integrated platform capable of executing a real-time data synchronization task scheduling optimization program. The system can execute a real-time data synchronization task scheduling optimization program. The method includes steps 101 to 105, as follows: Step 101: Obtain the first foreign key dependency features of each data table in the source database, generate the first execution priority sequence of the synchronization task based on the first foreign key dependency features, and determine the first preceding execution state of the synchronization task.

[0023] In this embodiment, the source database refers to the original database system from which the data to be synchronized can be a relational database such as MySQL, Oracle, or SQL Server. A data table refers to a table object in the source database that stores structured data, such as a user table, an order table, or a product table. The first foreign key dependency characteristic represents the reference relationship between data tables, used to describe the dependency relationship where the secondary table references the primary key of the primary table through a foreign key field; for example, the user ID field in the order table references the primary key ID of the user table. A synchronization task refers to the operation unit that synchronizes data from a specific data table in the source database to the target database. The first execution priority sequence represents the list of synchronization tasks to be executed in order based on the foreign key dependency relationship; for example, the user table synchronization task has a higher priority than the order table synchronization task. The first pre-execution state refers to the precondition state that a synchronization task needs to meet before execution, used to indicate the completion status of other tasks that the synchronization task depends on.

[0024] Specifically, the process of obtaining the first foreign key dependency features of each data table in the source database includes: sending a metadata read request to the source database, receiving and parsing the returned metadata information; extracting the constraint definition information of each data table from the metadata, including structural constraints such as primary keys and foreign keys; for each table, analyzing its foreign key definition, identifying the reference relationship with other tables, extracting the correspondence set of "referring table - referenced table", and forming a complete foreign key dependency graph. When generating the first execution priority sequence of synchronization tasks based on the first foreign key dependency features, a directed graph G=(V, E) is first constructed, where the vertex set V represents the data tables and the edge set E represents the foreign key dependency relationship. If table A depends on table B, there exists a directed edge from B to A; then, a topological sorting algorithm is performed on the directed graph to obtain a table sequence that conforms to the dependency relationship; finally, the table sequence T is mapped to the corresponding synchronization task sequence, i.e., the first execution priority sequence, where the tasks at the beginning of the sequence are executed first. When determining the first preceding execution state of a synchronous task, for each synchronous task, find all direct predecessor nodes in the dependency graph of its corresponding data table. The synchronous tasks corresponding to these predecessor nodes constitute the set of preceding tasks for the current task. Combine the execution states (not executed, executing, completed) of these preceding tasks to form the first preceding execution state of the current task, which is used for subsequent execution condition judgment.

[0025] In one possible implementation, the first foreign key dependency features of each data table in the source database are obtained, specifically including steps 1011-1012, as follows: Step 1011: Send a metadata read request to the source database and receive the metadata set returned by the source database; extract the constraint definition fields of each data table from the metadata set.

[0026] In this embodiment, the source database refers to the original database system from which data needs to be synchronized, such as relational databases like MySQL and Oracle. A metadata read request is a query instruction sent to the database to obtain database structure information. The metadata set represents a dataset returned by the database containing system-level information such as table structure, field types, and constraints; it typically includes a list of table names, column definitions, and index information. Constraint definition fields are specific fields in the metadata set that describe the constraints of the data table, including primary key definitions, foreign key relationships, and unique constraints.

[0027] Specifically, when sending a metadata read request to the source database, a dedicated SQL query statement is first constructed, using the corresponding system tables or views for different types of databases. For example, for MySQL, the INFORMATION_SCHEMA.KEY_COLUMN_USAGE table can be queried, and for Oracle, the ALL_CONSTRAINTS and ALL_CONS_COLUMNS views can be queried. Then, the query is executed through the database connection interface to obtain metadata information containing the relationships between tables. The query results returned by the source database are received, parsed, and structured and stored as a metadata collection. This collection typically contains multi-dimensional information such as table names, constraint names, constraint types, and related column names. Constraint records related to inter-table dependencies are filtered from the metadata collection, focusing primarily on FOREIGN KEY type constraints. Detailed information is extracted from each foreign key constraint record, including constraint name, the table containing the foreign key, the foreign key column name, the referenced table name, and the referenced column name, forming a constraint definition field set. The extracted constraint definition fields are grouped according to table names to form a constraint definition set by table, which is used for subsequent extraction of foreign key dependency features.

[0028] Step 1012: For each data table, extract the related fields from the constraint definition fields that correspond to the primary key of the related data table in the source database, and use the related fields as the first foreign key dependency feature of the data table.

[0029] In this embodiment, a data table refers to the basic unit for storing structured data in a relational database, such as a user table or an order table. Constraint definition fields represent metadata fields describing the relationships between tables, particularly the definition information of foreign key constraints. The primary key of a related data table refers to the primary key field in the table referenced by the foreign key (i.e., the parent or primary table), typically used as a field to uniquely identify records in that table. Related fields are fields in the current table that establish a reference relationship with the primary keys of other tables through foreign key constraints, representing the dependency relationship between the two tables. The first foreign key dependency feature represents the reference relationship characteristics between data tables, used to describe which tables the current table depends on and through which fields the dependency is established, forming a complete dependency graph. For example, the user ID field in the order table depends on the ID primary key in the user table.

[0030] Specifically, when extracting the related fields of the primary keys of the associated tables in the source database from the constraint definition fields of each data table, the process first involves filtering out all foreign key constraint records in the current table that are foreign key holders (child tables) from the constraint definition field set obtained in the previous step. For each foreign key constraint record, three key pieces of information are extracted: the foreign key column name, the referenced table name, and the referenced column name. Verification is made to confirm whether the referenced column is the primary key of the associated table, typically by checking if the column is in the primary key constraint of the associated table. The verified foreign key relationships are then organized into structured data, in the form of a mapping relationship of "current table.foreign key column → referenced table.primary key column". Multiple foreign key constraints that may exist in the same data table are aggregated to form a complete foreign key reference set for that table. Finally, this foreign key reference set is used as the first foreign key dependency feature of the data table. This feature fully describes the dependency network between the current table and other tables, including information on "which tables it depends on" and "which fields establish the dependency". These first foreign key dependency features will be used to construct the priority sequence of subsequent synchronization tasks, ensuring that data synchronization follows the inter-table dependency constraints.

[0031] In one possible implementation, a first execution priority sequence of the synchronization task is generated based on the first foreign key dependency feature, and a first pre-execution state of the synchronization task is determined, specifically including steps 1013-1015, as follows: Step 1013: Extract the primary table identifier and secondary table identifier from the first foreign key dependency feature; determine the synchronization task corresponding to the primary table identifier as the pre-execution item, and determine the synchronization task corresponding to the secondary table identifier as the post-execution item.

[0032] In this embodiment, the first foreign key dependency feature represents the reference relationship information between data tables, describing which tables reference other tables through foreign keys. The primary table identifier is a unique identifier for a data table referenced by other tables through a foreign key; for example, the user table is the primary table in a user-order relationship. The secondary table identifier is a unique identifier for a data table referenced by other tables through a foreign key; for example, the order table is the secondary table in a user-order relationship. Pre-execution items represent task units that need to be executed first in the synchronization task execution order, typically corresponding to data synchronization operations on the primary table. Subsequent execution items represent task units that need to be executed after the relevant pre-execution tasks in the synchronization task execution order, typically corresponding to data synchronization operations on the secondary table.

[0033] Specifically, when extracting the primary table identifier and secondary table identifier from the first foreign key dependency feature, each foreign key relationship record stored in the first foreign key dependency feature is first traversed; the referenced table name is extracted from each record as the primary table identifier, and the table name containing the foreign key is extracted as the secondary table identifier; after analyzing all foreign key relationships, all primary table identifiers and secondary table identifiers are summarized, duplicates are removed, and unique primary table lists and secondary table lists are formed. When the synchronization task corresponding to the primary table identifier is determined as a pre-execution item, a corresponding synchronization task object is created for each primary table according to the primary table list, and its task type is marked as "pre-execution item"; similarly, when the synchronization task corresponding to the secondary table identifier is determined as a post-execution item, a corresponding synchronization task object is created for each secondary table according to the secondary table list, and its task type is marked as "post-execution item"; after completion, two task sets are formed: a pre-execution task set and a post-execution task set.

[0034] Step 1014: Arrange the execution order of the pre-execution items and post-execution items to obtain the first execution priority sequence; in the first execution priority sequence, obtain the synchronization task corresponding to the sub-table identifier, and find the target synchronization task corresponding to each master table identifier associated with the synchronization task corresponding to the sub-table identifier.

[0035] In this embodiment, the pre-execution item represents the synchronization task that needs to be executed first, usually corresponding to the synchronization of the master table data. The post-execution item represents the synchronization task that needs to be executed only after the relevant pre-execution task is completed, usually corresponding to the synchronization of the slave table data. Execution order refers to the ranking of the execution order of synchronization tasks in the entire synchronization process. The first execution priority sequence represents a complete list of synchronization task execution order determined by foreign key dependencies, determining which tables are synchronized first and which are synchronized later. Slave table identifier refers to the unique identification symbol of a data table that references other tables through foreign keys, such as an order table. Master table identifier refers to the unique identification symbol of a data table that is referenced by other tables, such as a user table. Target synchronization task refers to the specific synchronization operation unit executed for a specific data table, including execution parameters such as table name, synchronization mode, and data range.

[0036] Specifically, when arranging the execution order of preceding and subsequent execution items, all preceding execution items (synchronization tasks corresponding to the main table) are first placed at the beginning of the sequence. If there are dependencies between preceding execution items, their relative order is determined using a topological sorting algorithm. Then, all subsequent execution items (synchronization tasks corresponding to the slave table) are placed at the end of the sequence. Similarly, if there are dependencies between subsequent execution items, their relative order is also determined using topological sorting. The preceding and subsequent execution item sequences are merged to form a complete first execution priority sequence. When retrieving the synchronization task corresponding to the slave table identifier in the first execution priority sequence, each task in the first execution priority sequence is traversed, and tasks with the task type "subsequent execution item" are selected, i.e., the synchronization tasks corresponding to the slave table. For each synchronization task corresponding to a slave table, based on the association relationship in the first foreign key dependency feature, all main tables referenced by the slave table through foreign keys are found. Based on these main table identifiers, the corresponding synchronization task, i.e., the target synchronization task, is found in the synchronization task set. A mapping relationship is established between the slave table synchronization task and its dependent main table target synchronization task for subsequent execution status checks.

[0037] Step 1015: Extract the execution completion status of the target synchronization task corresponding to each master table identifier, and determine the execution completion status as the first preceding execution status of the synchronization task corresponding to the slave table identifier.

[0038] In this embodiment, the primary table identifier represents the unique identifier of the data table referenced by the foreign key, such as the product table identifier in an order-product relationship. The target synchronization task refers to a data synchronization operation unit created for a specific data table, including parameters such as table name, synchronization scope, and execution mode. The execution completion status refers to the progress marker of the synchronization task during execution, used to indicate whether the task has been completed, is being executed, or has not yet started; it is typically represented by status values ​​such as "not started," "in execution," and "completed." The secondary table identifier represents the unique identifier of the data table containing the foreign key, such as the order table identifier in an order-product relationship. The first pre-execution state represents the set of dependency conditions that the synchronization task needs to satisfy before execution, used to ensure that the tasks are executed in the correct order and guarantee data consistency.

[0039] Specifically, when extracting the execution completion status of the target synchronization task corresponding to each master table identifier, the process first iterates through all master table identifiers associated with the slave table synchronization task. For each master table identifier, the corresponding target synchronization task record is searched in the synchronization task status table. The current execution status field value is read from the target synchronization task record, which typically includes statuses such as "not started," "in execution," "completed," or "failed." The execution status values ​​of all master table target synchronization tasks are collected into a status set. When the execution completion status is determined as the first preceding execution status of the synchronization task corresponding to the slave table identifier, the set of master table target synchronization task execution statuses collected in the previous steps is treated as a whole and assigned to the preceding execution status attribute of the slave table synchronization task. A preceding status check condition is added to the metadata of the slave table synchronization task, specifying that the slave table synchronization task is allowed to start execution only when the status of all dependent master table tasks is "completed." A preceding execution status listener is created, which automatically re-evaluates whether the execution conditions of the slave table task are met when the status of any related master table task changes.

[0040] Step 102: Extract the resource requirement identifier of the synchronization task, obtain the current resource quantity of the target node, and distribute the synchronization task to the target node according to the resource requirement identifier and the current resource quantity to obtain the task distribution plan.

[0041] In this embodiment, the resource requirement identifier refers to the expected computing resource indicators consumed by the synchronization task during execution, including processor utilization and memory requirements. The target node represents the computing node executing the synchronization task, which can be a physical server, virtual machine, or container instance. The current resource quantity refers to the amount of computing resources available on the target node for allocation to new tasks. The task distribution plan is a mapping table between synchronization tasks and execution nodes, defining the specific arrangements of which tasks will be executed on which nodes.

[0042] Specifically, extracting the resource requirement identifier for a synchronization task includes: analyzing the structural characteristics of the source data table, such as the number of records, field types, and average row size; estimating the total amount of data involved in the synchronization process; calculating the memory and CPU utilization required to process a unit amount of data based on historical synchronization performance data; and obtaining the resource requirement identifier for the synchronization task, representing the data volume, memory requirements, and processor requirements. When obtaining the current resource quantity of the target node, the resource status of each candidate node is obtained through the system monitoring interface, including total memory capacity, used memory, total number of CPU cores, and current CPU utilization; the available memory and processor idle rate of each node are calculated; and these resource indicators are combined to form the current resource quantity of the node. When distributing the synchronization task to the target node based on the resource requirement identifier and the current resource quantity, for each synchronization task and candidate node, it is determined whether the resource requirement conditions are met; from the set of candidate nodes that meet the conditions, the node with the highest resource surplus is selected as the target node for the task; and a mapping relationship between the task and the target node is established to form a complete task distribution plan.

[0043] In one possible implementation, the resource requirement identifier of the synchronization task is extracted, the current resource quantity of the target node is obtained, and the synchronization task is distributed to the target node according to the resource requirement identifier and the current resource quantity to obtain a task distribution plan. Specifically, this includes steps 1021-1023, as follows: Step 1021: Obtain the estimated data processing volume of the synchronization task and use the estimated data processing volume as the resource requirement identifier.

[0044] In this embodiment, the estimated data processing volume represents the expected size of data that the synchronization task needs to process, typically measured in bytes, rows, or records, such as "500MB" or "1 million rows of records." The estimated data processing volume reflects the resource consumption potential of the synchronization task; a larger data volume usually means more computing and storage resources are required. Resource requirement identifiers are numerical indicators used to quantify the computing resources required by the synchronization task, serving as a reference for task scheduling and allocation. These typically include requirements for CPU, memory, and network bandwidth. A synchronization task refers to the operation unit that copies data from the source database to the target database, including execution parameters such as table names, synchronization methods, and data ranges.

[0045] Specifically, when obtaining the estimated data processing volume of a synchronization task, the process first involves querying the statistical information of the target table in the source database, including the total number of rows, average row length, and index size. Then, the original data volume of the table is calculated based on the total number of rows and row size using the formula: Original data volume = Total number of rows × Average row length. Considering the overhead of data transmission and conversion, an amplification factor k (usually k > 1) is introduced, resulting in the adjusted data processing volume = Original data volume × k. If the synchronization task contains multiple tables, the estimated processing volume of each table is calculated separately and summed to obtain the total estimated data processing volume of the task. Finally, the calculated estimated data processing volume value is directly used as the resource requirement identifier for the synchronization task for subsequent resource matching and scheduling decisions.

[0046] Step 1022: Obtain the remaining memory and processor idle rate of multiple candidate nodes, and determine the current resource quantity based on the remaining memory and processor idle rate; select candidate nodes whose resource requirements are less than the current resource quantity as target nodes.

[0047] In this embodiment, the current resource quantity represents the total computing resources available for a system node to execute synchronous tasks at a specific moment, typically determined based on a comprehensive assessment of memory and CPU resource status. Remaining memory refers to the amount of unused memory space on a computing node, usually measured in bytes or megabytes. Processor idle rate refers to the proportion of time a CPU core is not occupied, usually expressed as a percentage, such as "70% idle" indicating that 70% of the CPU's processing power is available for new tasks. Candidate nodes refer to computing server instances that may be used to execute synchronous tasks, including physical servers or virtual machine instances. Target nodes are the computing nodes ultimately selected to execute a specific synchronous task, requiring them to meet the task's resource requirements. The resource requirement identifier indicates the amount of computing resources required for the synchronous task, used for matching and comparison with the node's available resources.

[0048] Specifically, when obtaining the remaining memory and processor idle rate of multiple candidate nodes, the resource usage of each candidate node is first collected in real time through the system monitoring interface or agent program. The formula for remaining memory is: Remaining Memory = Total Node Memory - Used Memory, in MB or GB. The formula for processor idle rate is: Processor Idle Rate = (1 - CPU Utilization), expressed as a decimal or percentage between 0 and 1. Resource data from all candidate nodes is collected to form a resource status list. When determining the current resource quantity based on remaining memory and processor idle rate, a weighted calculation method is used, with the formula: Current Resource Quantity = α × Remaining Memory + β × Processor Idle Rate × Number of Processor Cores, where α and β are weighting coefficients adjusted according to the type of synchronization task. The current resource quantity value is calculated for each candidate node, forming a node resource quantity list. When selecting candidate nodes whose resource requirement identifier is less than the current resource quantity as target nodes, all candidate nodes are traversed, and resource sufficiency is judged for each node; nodes that meet the condition "resource requirement identifier < current resource quantity" are marked as qualified nodes; the node with the largest current resource quantity is selected from the qualified nodes as the target node; if multiple nodes have similar current resource quantities, a load balancing factor can be introduced for secondary screening; the finally determined target node is recorded in the task distribution mapping table to prepare for generating the task distribution plan.

[0049] Step 1023: Generate a task distribution plan based on the matching results between the synchronization task and the target node.

[0050] In this embodiment, the task distribution plan represents the allocation scheme between synchronization tasks and execution nodes, describing which synchronization task will be executed on which target node, and the execution order. A synchronization task refers to a specific operational unit that copies data from a source database to a target database, including parameters such as table name, synchronization scope, and execution mode. A target node refers to the computing server selected to execute the synchronization task, possessing sufficient resources to support task execution. The matching result refers to the matching relationship between synchronization tasks and target nodes, indicating a successful pairing of resource requirements and resource supply.

[0051] Specifically, when generating a task distribution plan based on the matching results between synchronous tasks and target nodes, the process first integrates all the matching relationships between synchronous tasks and target nodes obtained in the previous steps to form a complete matching mapping table. This mapping table is then structured and converted into a standardized task distribution plan format, which typically includes fields such as task ID, task type, target node ID, resource requirements, and estimated execution time. A unique execution identifier is assigned to each task for subsequent task tracking and status management. Based on resource requirements and dependencies, the execution order of multiple tasks on the same node is determined, generating a task queue within the node. The distribution of tasks and nodes is checked for balance, and adjustments are made as necessary to avoid overloading some nodes while leaving others idle. The final task distribution plan is serialized into a specific format, such as JSON or XML, for easy system parsing and execution. The integrity of the generated plan is verified to ensure that all tasks have corresponding target nodes and that resource allocation meets execution requirements. Finally, the verified task distribution plan is submitted to the task scheduling system, ready to execute synchronous tasks according to the plan.

[0052] Step 103: Execute synchronization tasks on the target node according to the task distribution plan. When the running load of the target node is greater than the preset load threshold, migrate the unexecuted synchronization tasks on the target node to the standby node and record the task receiving status of the standby node.

[0053] In this embodiment, the task distribution plan represents the set of correspondences between synchronization tasks and execution nodes. A target node is a computing node selected to execute a specific synchronization task. The runtime load represents the resource consumption of a node during task execution, typically measured by metrics such as CPU utilization and memory usage. The preset load threshold refers to the maximum allowable load level predefined by the system. Unexecuted synchronization tasks refer to synchronization operations assigned to nodes but not yet started or completed. A standby node represents a redundant computing node that can take over tasks when the primary node is overloaded. The task reception status refers to the status information returned by the standby node after receiving the migrated task.

[0054] Specifically, the process of executing synchronization tasks on target nodes according to the task distribution plan includes: sending task execution instructions to each target node, the instructions containing parameters such as task identifier, source table information, target table information, and synchronization mode; after receiving the instructions, the target node starts the corresponding synchronization process, establishes connections with the source database and the target database, and determines whether the execution conditions are met based on the first pre-order execution status; when the pre-order conditions are met, data reading, transformation, and writing operations are started to achieve synchronous transmission of table data. When the target node's workload exceeds a preset load threshold, the system monitoring module collects real-time metrics such as CPU utilization, memory utilization, and disk I / O utilization. It calculates a comprehensive load metric, and when this metric exceeds the preset load threshold, it triggers a task migration mechanism, sending a pause command to the target node. Upon receiving the pause command, the target node stops receiving new synchronization operations and records its current execution position as a task execution progress indicator. It then extracts a set of unexecuted synchronization tasks from the task queue and selects a resource-sufficient backup node from the available node pool, sending the task set and its execution context information to the backup node. Upon receiving the tasks, the backup node verifies resource conditions and initializes the execution environment. Upon success, it returns a successful reception response containing the node identifier, reception timestamp, and task list. The system records this response information as the task reception status for subsequent task execution monitoring and recovery.

[0055] Step 104: Obtain the operation change logs generated by the standby node and the target node during the execution of the synchronization task, and compare the execution time sequence of the operation change logs with the sorting result of the first execution priority sequence.

[0056] In this embodiment, the operation change log represents data operation records generated during the execution of the synchronization task, including information such as table identifier, operation type, and execution timestamp. The execution time sequence refers to the sequence obtained by sorting the table operations according to the timestamps in the operation change log, reflecting the actual time order of the synchronization operations. The sorting result refers to the sequence formed by extracting the corresponding data table identifiers according to the first execution priority sequence, used for comparison with the execution time sequence.

[0057] Specifically, obtaining the operation change logs generated by the backup and target nodes during the synchronization task includes: sending a log retrieval request to each participating node; after receiving the request, each node extracts operation records related to the current synchronization session from its local log storage, including fields such as the data table name, operation type, operation start time, operation completion time, and number of rows affected; each node returns the extracted log records to the central coordinator, forming a complete set of operation change logs. When comparing the execution time sequence of the operation change logs with the sorting result of the first execution priority sequence, firstly, the completion timestamp of the first operation on each table and the corresponding table identifier are extracted from the operation change log set to form a set of tuples; then, the set is sorted in ascending order according to the timestamps to obtain a sequence of table identifiers arranged by actual execution time, which is the execution time sequence; next, the corresponding data table identifiers are extracted sequentially from the first execution priority sequence to form a table identifier sequence, which is the sorting result; finally, the two sequences are compared bit by bit to check if they are completely consistent. If they are all equal, it means that the actual execution order is consistent with the expectation; otherwise, it means that there is a deviation in the execution order.

[0058] In one possible implementation, comparing the execution time sequence of the operation change log with the sorting result of the first execution priority sequence specifically includes steps 1041-1043, as follows: Step 1041: Extract the data table identifier and operation completion timestamp from the operation change log; sort the data table identifiers according to the operation completion timestamp to obtain the execution time sequence of the operation change log.

[0059] In this embodiment, the operation change log represents a detailed record of all data change operations recorded by the database system, including information on operations such as insert, delete, and modify, such as MySQL's binlog and Oracle's redo log. A table identifier is a unique identifier that can identify a data table, usually the table name or table ID. An operation completion timestamp refers to the precise time point at which the data change operation was completed, typically recorded with millisecond precision. The execution time sequence represents an ordered sequence formed by arranging the operations in chronological order, reflecting the temporal relationship of the actual data changes.

[0060] Specifically, when extracting table identifiers and operation completion timestamps from the operation change log, the database system's change log file is first parsed, such as using a binlog parsing tool to read MySQL's binary log. Metadata information for each change record is extracted from the log file, including the operation type (INSERT / UPDATE / DELETE), the affected table name, and execution time. The table name is extracted as the table identifier. Simultaneously, the time information for each operation record is extracted and converted into a unified timestamp format, accurate to the millisecond level. The extracted table identifiers and timestamps are combined into a set of tuples. The tuple set is then sorted in ascending order using a time-series sorting algorithm based on the operation completion timestamp, ensuring that earlier operations are listed first. After sorting, the table identifiers are extracted in chronological order to form an ordered sequence. This ordered sequence represents the execution time sequence of the operation change log, preserving the chronological order of the actual database operations.

[0061] Step 1042: Extract the corresponding data table identifiers in sequence according to the execution priority of each synchronous task in the first execution priority sequence, and generate the sorting results.

[0062] In this embodiment, the first execution priority sequence represents a list of synchronization task execution orders determined based on the dependencies between data tables, serving as the basis for task scheduling during data synchronization. Execution priority refers to the priority level of a synchronization task within the overall synchronization process, typically represented numerically; a smaller value indicates a higher priority and requires earlier execution. A data table identifier is an identifier that uniquely identifies a data table, usually a table name or table ID. The sorting result refers to the sequence of data table identifiers rearranged according to execution priority, reflecting the theoretically optimal data synchronization order.

[0063] Specifically, when extracting the corresponding data table identifiers according to the execution priority of each synchronization task in the first execution priority sequence, the complete first execution priority sequence is first obtained. This sequence contains all tasks that need to be synchronized and their priority information. It is then checked whether the task objects in the sequence are sorted by priority. If not, priority is used as the key to sort them, ensuring that higher-priority tasks are listed first. From the sorted task sequence, the data table identifiers associated with each synchronization task are extracted sequentially according to their position. Specifically, for each task object in the sequence, its table identifier attribute is accessed to obtain the associated data table identifier. All extracted data table identifiers are arranged sequentially according to the original task order in the priority sequence, forming a new ordered array. This array is the sorting result, where each element represents a data table to be synchronized, and the order of elements in the array reflects the ideal synchronization order determined based on the inter-table dependencies.

[0064] Step 1043: Compare the execution time sequence of the operation change log with the data table identifier of the same position in the sorting result item by item to obtain the comparison result.

[0065] In this embodiment, the execution time sequence of the operation change log represents a sequence of data table identifiers arranged chronologically according to the operation time, reflecting the actual time order of data changes. The sorting result represents a sequence of data table identifiers sorted according to execution priority, reflecting the ideal synchronization order determined based on inter-table dependencies. "Same position" refers to elements at the same index position in two different sequences, such as the first element, second element, etc., of two sequences. The comparison result represents a dataset generated by comparing the matching of elements at corresponding positions in two sequences, used to evaluate the consistency between the actual operation order and the ideal synchronization order.

[0066] Specifically, when comparing the execution time sequence of the operation change log with the data table identifiers at the same position in the sorting results item by item, the effective comparison length of the two sequences is first determined, and the smaller value of the two sequence lengths is taken as the comparison range; a comparison result array with the same comparison length is created to record the comparison status of each position; starting from the index, it is traversed sequentially to the end position, and each index position is compared; for each index, it is checked whether the execution time sequence and the sorting result represent the same data table identifier at that position; if they are the same, it is recorded as "match" in the corresponding position of the comparison result array, otherwise it is recorded as "mismatch"; at the same time, the specific difference information of each mismatch is recorded, including the position index, expected value and actual value; after completing the comparison of all positions, the overall matching rate is calculated, that is, the percentage of the number of matching items to the total number of comparison items; finally, a comparison result object is generated, which includes a position matching status array, a list of mismatch details and overall matching statistics.

[0067] Step 105: When the execution time sequence is the same as the sorting result, generate a synchronization success confirmation signal and update the synchronization completion flag of the source database according to the synchronization success confirmation signal; when the first foreign key dependency feature changes, generate a second execution priority sequence according to the changed second foreign key dependency feature and determine the second preceding execution state of each synchronization task.

[0068] In this embodiment, the synchronization success confirmation signal refers to the success identification information generated after verifying the correctness of the synchronization task execution result. The synchronization completion flag represents a flag bit in the source database used to identify the synchronization status of the data table. The second foreign key dependency feature refers to the new dependency relationship feature after the foreign key relationship has changed. The second execution priority sequence represents the task execution order regenerated according to the updated foreign key dependency relationship. The second pre-execution state refers to the task precondition state determined based on the new dependency relationship.

[0069] Specifically, when the execution time sequence and sorting result are the same, the transaction commit identifier is first retrieved from the operation change log, including the transaction ID, commit timestamp, and commit status code corresponding to the last operation of each table; the extracted transaction commit identifier is bound to the table name of the corresponding data table to form an association pair; when the commit status of all tables is successful, a synchronization success confirmation signal containing a success status code, synchronization session ID, a list of synchronized tables, and a completion timestamp is generated; this signal is sent to the control terminal of the source database through the network interface; after receiving the confirmation signal, the control terminal parses the signal content and identifies the list of synchronized tables; an update operation is performed on each table, modifying the status field in the table's metadata to the synchronization completion flag. When the first foreign key dependency feature changes, the latest table structure definition file of the source database is read through a periodic check mechanism or change notification mechanism; the current foreign key constraint definition is extracted from the table structure definition to form the current foreign key dependency feature; the current foreign key dependency feature is compared with the previously recorded first foreign key dependency feature using a comparison algorithm to detect whether there is a difference; if there is a difference, the current foreign key dependency feature is set as the second foreign key dependency feature; based on the second foreign key dependency feature, a new execution priority sequence is generated using a topological sorting method, i.e., the second execution priority sequence; based on the new dependency graph, the set of preceding tasks for each synchronous task is recalculated, and the corresponding second preceding execution state is determined for use in determining the execution conditions of subsequent synchronous tasks.

[0070] In one possible implementation, when the execution time sequence is the same as the sorting result, a synchronization success confirmation signal is generated, and the synchronization completion flag of the source database is updated according to the synchronization success confirmation signal, specifically including steps 1051-1052, as follows: Step 1051: When the execution time sequence is the same as the sorting result, extract the transaction commit identifier from the operation change log; bind the transaction commit identifier to the table name of the data table, and generate a synchronization success confirmation signal.

[0071] In this embodiment, the execution time sequence represents an ordered sequence arranged according to the actual occurrence time of database operations, reflecting the true time order of change operations in the database. The sorting result represents the ideal synchronization order determined based on inter-table dependencies, which is derived from the execution priority sequence generated based on foreign key dependencies. The transaction commit identifier is a unique identifier for a successfully committed database transaction, typically containing information such as the transaction ID and commit timestamp, used to confirm that a specific transaction has been committed. The table name is a unique name identifier for the data table in the database, such as "users" or "orders". The synchronization success confirmation signal is a notification message indicating that the synchronization operation of a specific data table has been successfully completed, containing the table name and transaction information, used to notify the source system to update the synchronization status.

[0072] Specifically, when the execution time sequence and sorting result are the same, the transaction commit identifier in the operation change log is extracted. First, the elements of the two sequences, execution time sequence and sorting result, are compared to see if they are completely identical, including element values ​​and order. After confirming that the two sequences are completely identical, the transaction information corresponding to each data operation is extracted from the original operation change log, including transaction ID, commit timestamp, operation type, etc. Identifiers indicating successful transaction commit are filtered from the transaction information, such as the "COMMIT" record in the log and its related metadata. For each successfully committed transaction, its unique identifier information is extracted to form a set of transaction commit identifiers. When binding the transaction commit identifier with the table name, an association mapping is constructed for each table, with the table name as the key and the corresponding transaction commit identifier as the value. If a table involves multiple transactions, there may be multiple transaction identifiers. In this case, the latest transaction commit identifier is usually selected for binding. The binding relationship between the table name and the transaction commit identifier is serialized into a specific format, such as a JSON object or XML structure. A complete list containing successfully synchronized tables is constructed, and transaction commit details corresponding to each table are attached. Finally, a structured synchronization success confirmation signal is generated, which contains the list of successfully synchronized tables, transaction commit information for each table, synchronization time, and other metadata.

[0073] Step 1052: Send a synchronization success confirmation signal to the control terminal of the source database so that the control terminal modifies the corresponding status field of the data table to the synchronization completion flag.

[0074] In this embodiment, the synchronization success confirmation signal indicates that a specific data table has completed the synchronization operation, and includes core information such as the table name and transaction identifier. The source database refers to the original database system from which data needs to be synchronized to the target database; it is the source of the data. The control unit refers to the control component that manages the synchronization status of the source database, responsible for tracking and recording the synchronization progress. Status fields are specific fields in the source database used to mark the synchronization status of data tables, typically located in metadata tables or system tables. Synchronization completion flags are status values ​​indicating that the data table synchronization operation has been successfully completed, such as "COMPLETED" or "SYNCED".

[0075] Specifically, when sending a synchronization success confirmation signal to the source database control terminal, the signal is first formatted into a standard message format, typically JSON or XML, ensuring it includes key information such as table name and transaction identifier. A communication connection is established with the source database control terminal, which can be achieved through an HTTP interface, message queue, or direct database connection. The formatted synchronization success confirmation signal is then sent to the designated receiving interface or message queue on the control terminal. Appropriate message delivery parameters are set, such as message priority, timeout, and retry strategy, to ensure reliable message delivery. A message delivery confirmation mechanism is implemented, awaiting a return confirmation from the control terminal. The message sending status and the control terminal's response are recorded for subsequent tracking and troubleshooting. After receiving the synchronization success confirmation signal, the control end of the source database parses the message content and extracts key information such as table name and transaction identifier. Based on the table name, it locates the corresponding synchronization status table or system table in the source database. It then performs a status update operation, changing the corresponding status field of the data table from its previous state (e.g., "SYNCING" or "PENDING") to a synchronization completion marker (e.g., "COMPLETED"). This update operation is typically implemented using SQL statements, such as "UPDATE sync_status SET status = 'COMPLETED' WHERE table_name = 'table_name'". Simultaneously, it updates other relevant fields in the status table, such as the synchronization completion time and the last synchronized transaction ID. Finally, it logs the status update operation, including update time, operator, and status change details, for subsequent tracking and auditing.

[0076] In the above embodiments, basic data synchronization task scheduling functions are achieved through task dependency analysis and resource matching. To further improve the stability and continuity of the system under load fluctuation conditions, and to establish a dynamic task migration mechanism between the target node and the backup node, this application also provides a real-time load-aware task migration and recovery method. This method monitors the changes in the computing resource utilization and processing capacity of the target node in real time, constructs a task execution state saving and migration protocol, and performs seamless task switching and route update processing between nodes, enabling the system to maintain the continuity and integrity of data synchronization tasks in the face of sudden load increases or node resource constraints. By establishing a collaborative mechanism between the primary and backup nodes, a balance between high availability of synchronization tasks and optimized overall system throughput is achieved, ensuring that the data synchronization process is not interrupted due to single-node performance fluctuations. The following section combines... Figure 2 The following describes a real-time data synchronization task scheduling optimization method in an embodiment of this application: Please see Figure 2 This is a flowchart illustrating a real-time data synchronization task scheduling optimization method in an embodiment of this application.

[0077] Step 201: When the running load of the target node exceeds the preset load threshold, send a task pause command to the target node.

[0078] In this embodiment, the operating load represents the computing resource usage of the target node, typically including comprehensive indicators such as CPU utilization, memory usage ratio, and disk I / O pressure. It is a quantitative indicator of the node's processing capacity saturation. The target node refers to the computing server assigned to execute the synchronization task, undertaking the main computing work for data synchronization. The preset load threshold refers to the upper limit of node load pre-set by the system. When the node's operating load exceeds this threshold, it indicates that the node is in a high-load operating state, such as CPU utilization exceeding 85% or memory usage exceeding 90%. The task pause command is a control command sent to the target node requesting the pause of the currently executing synchronization task, used to prevent node overload from causing system instability.

[0079] Specifically, when the target node's workload exceeds a preset load threshold, a task pause command is sent to the target node. First, the system monitoring interface collects real-time data on the target node's resource usage, including CPU utilization, memory usage, network bandwidth utilization, and other metrics. Based on a preset calculation model, these resource metrics are integrated into a comprehensive load metric, typically using a weighted average. The calculated workload is compared to the system's preset load threshold to determine if it exceeds the threshold. Once it's confirmed that the workload exceeds the preset load threshold, a standard-format task pause command is constructed, containing necessary information such as the operation type (pause), target task ID, and timestamp. A suitable communication method, such as HTTP request, message queue, or Remote Procedure Call (RPC), is selected to send the task pause command to the target node's task control interface. A timeout and retry mechanism are set for command transmission to ensure reliable delivery even under unstable network conditions. The time, content, and target node of the command transmission are recorded to prepare for subsequent task migration operations. Simultaneously, response monitoring is initiated, awaiting the target node's confirmation of the pause command.

[0080] Step 202: Obtain the task execution progress indicator when the target node receives the task pause instruction; extract the unexecuted synchronous tasks based on the task execution progress indicator, and send the unexecuted synchronous tasks to the standby node.

[0081] In this embodiment, the task execution progress identifier refers to marker information indicating the current completion status of the synchronization task, typically including the amount of data processed, the current processing position, and the completion percentage, used to accurately record the real-time status of the synchronization task. The target node represents a computing server currently executing a synchronization task but needing to pause the task due to excessive load. The task pause command is a control command requesting the termination or suspension of an ongoing synchronization task. Unexecuted synchronization tasks refer to data synchronization operations that have been assigned but not yet started or interrupted during execution, requiring reallocation to other nodes for completion. Backup nodes are computing servers reserved in the system that can take over tasks from other nodes, typically maintaining a low resource utilization rate to cope with emergency task migration needs.

[0082] Specifically, when obtaining the task execution progress identifier when the target node receives the task pause command, the system first receives the status report returned by the target node in response to the task pause command. This report contains the current execution status of each synchronous task. The system then extracts the execution progress identifier for each synchronous task from the status report, including the current data processing location, the number of completed records, and processing time. The extracted execution progress identifiers are then categorized and organized according to task ID to form a task-progress mapping table. When extracting unexecuted synchronous tasks based on their execution progress identifiers, the system analyzes the execution progress of each synchronous task to determine its completion status. For tasks with a progress of 0%, they are marked as "not started." For tasks with a progress greater than 0% but less than 100%, their breakpoint location information is obtained to prepare for subsequent breakpoint resumption. For tasks already marked as "not started," their complete task description information is directly added to the list of tasks to be migrated. For interrupted tasks, a task description containing breakpoint information is constructed to ensure that execution can continue from the breakpoint location on the standby node. Finally, all unexecuted or interrupted synchronous tasks are summarized to form a complete task migration list. When sending unexecuted synchronization tasks to standby nodes, first select a standby node with low current load and sufficient resources from the system; construct a task distribution message, including necessary parameters such as task description, priority, and breakpoint information; send the task distribution message to the selected standby node through the system's task distribution interface or message queue; set a timeout and retry mechanism for task sending to ensure that task information can be reliably delivered; record detailed information about task migration, including the original node ID, the target standby node ID, the migration time, and the migration task list.

[0083] Step 203: When the standby node returns a successful reception response, record the successful reception response as the task reception status.

[0084] In this embodiment, a successful response indicates that the standby node has successfully received and acknowledged the reply message confirming that it can handle the migration task. This typically includes information such as the reception time, a list of task IDs, and a node identifier. A standby node refers to a pre-reserved computing server in the system that can take over tasks from the target node that needs to pause due to excessive load, possessing the same processing capabilities as the original target node. Task reception status refers to the status information recorded by the system during the task migration process, used to track whether the task migration was successfully completed, including status values ​​such as "sent," "received," and "reception failed."

[0085] Specifically, when the standby node returns a successful reception response, this response is recorded as the task reception status. First, the system receives the response message from the standby node, indicating that it has successfully received the migrated synchronization task. The response message content is parsed to extract key information such as the list of received task IDs, node identifiers, and reception timestamps. The integrity and validity of the response message are verified to ensure that all migration tasks have been acknowledged and received by the standby node. The content of the successful reception response message is converted into a standard task reception status format, including fields such as task ID, original node, standby node, migration time, and reception status. The system's task status management table is updated, changing the status of the relevant tasks from "Migration in Progress" to "Received." Complete status change information is recorded in the task scheduling system log, including the time of the status change, operation type, and involved nodes. The task reception status information is synchronized to the system monitoring panel, allowing administrators to monitor the task migration status in real time. For particularly important tasks, a status change notification may be triggered, notifying relevant personnel via email, SMS, etc., that the task has been successfully migrated to the standby node. After completing the status recording, the system continues to monitor the execution status of tasks on the standby node to ensure successful task completion.

[0086] In one possible implementation, after recording the successful reception response as the task reception status, the process further includes steps 2031-2033, as follows: Step 2031: Extract the standby node identifier from the task receiving status.

[0087] In this embodiment, the task reception status represents the processing status record of the backup node receiving the migration task, including information such as reception time, task list, and processing result. The backup node identifier is a unique identifier for a specific backup node in a distributed system, typically composed of node ID, IP address, hostname, etc., such as "backup-node-003" or "192.168.1.105". The backup node identifier is used to accurately locate and access specific backup computing resources in the system, ensuring that tasks can be correctly routed and executed.

[0088] Specifically, when extracting the backup node identifier from the task reception status, the system first accesses the task reception status record stored in the system, which is usually stored in the task status management table or task status database; it then parses the data structure of the task reception status to locate the field or attribute containing backup node information; it extracts the complete backup node identifier information from this field, which may include node ID, IP address, port number, etc.; it verifies the format and validity of the extracted backup node identifier to ensure that it conforms to the identifier specification defined by the system; if the task reception status contains multiple backup node identifiers (e.g., the task is distributed across multiple backup nodes), it is necessary to extract and construct a backup node identifier list one by one; it then converts the extracted backup node identifiers into the standard format used internally by the system for subsequent routing update operations; finally, it stores the formatted backup node identifiers in a temporary variable or cache, ready for use in the next step of node routing information update.

[0089] Step 2032: Update the node routing information in the task distribution plan based on the standby node identifier.

[0090] In this embodiment, node routing information represents the network path configuration for task distribution in the system, recording the mapping relationship of which specific nodes each synchronization task should be sent to for execution. Node routing information includes key data such as source node identifier, target node identifier, task type, and network path, and is the basic configuration for the system to correctly distribute and execute tasks. The task distribution plan refers to the complete execution scheme formulated by the system for performing data synchronization, including information such as task list, execution order, resource allocation, and node allocation. The standby node identifier is an identifier used to uniquely identify the standby computing node that will take over the task from the original target node, such as a node ID or network address.

[0091] Specifically, when updating node routing information in the task distribution plan based on the standby node identifier, the currently active task distribution plan is first loaded. This plan contains all synchronization tasks and their corresponding execution node information. The task entries that need updating are located in the task distribution plan; these are typically tasks previously handled by the original target node but now migrated to the standby node. The target execution node field in these task entries is modified, changing it from the original target node identifier to the standby node identifier. Simultaneously, relevant routing parameters, such as network address, port number, and connection protocol, are updated. If the system uses a hierarchical routing structure, the forwarding tables of intermediate routing nodes also need to be updated accordingly. To ensure the consistency of routing updates, a transaction mechanism is used for the update operation, ensuring that all relevant routing items are either updated successfully or rolled back completely. After the update is completed, a routing change log is generated, recording the routing information before and after the change for subsequent tracking and troubleshooting. Finally, the updated task distribution plan is synchronized to the system's configuration center or distributed to each control node to ensure that the entire system uses the latest routing information.

[0092] Step 2033: Send a task recovery command to the standby node so that the standby node can continue to execute the unexecuted synchronization task based on the updated node routing information.

[0093] In this embodiment, the task recovery instruction refers to the control command sent to the standby node to start or continue the execution of a previously paused synchronization task, including the task ID, execution parameters, breakpoint information, etc. The standby node represents the computing server that takes over the original target node's task and has the capability to execute synchronization tasks. Node routing information refers to the task execution path configuration maintained in the system, describing the data flow and processing nodes during the data synchronization process. Unexecuted synchronization tasks refer to data synchronization operations that were paused by the original target node due to excessive load and have not yet been completed, including tasks that have not started at all and tasks whose execution has been interrupted.

[0094] Specifically, when sending a task recovery command to the standby node, a standard format task recovery command is first constructed, which includes key information such as operation type (recovery), task ID list, task parameters, and execution priority. For previously interrupted tasks, breakpoint recovery information is added to the command, including the last execution location and the amount of data processed. Updated node routing information is appended as a parameter to the task recovery command to ensure that the standby node can correctly access the relevant data source and target location. Control parameters such as the command's validity period, importance level, and execution timeout are set. An appropriate communication protocol and channel, such as HTTP request, message queue, or RPC call, is selected to send the constructed task recovery command to the standby node's command receiving interface. A reliability guarantee mechanism is implemented during the sending process, including acknowledgment receipt, failure retry, and timeout handling. The time, content, and receiving node of the command are recorded to form a complete operation log. At the same time, a monitoring program is started to track the status of the standby node's receipt and execution of commands, providing timely responses to possible anomalies. After confirming that the standby node has successfully received the command, the status of the relevant task in the system is updated to "recovering," indicating that the task has changed from a paused state to being executed on the standby node.

[0095] The following describes a real-time data synchronization task scheduling optimization system according to an embodiment of the present invention from the perspective of hardware processing. Please refer to [link / reference needed]. Figure 3 This is a schematic diagram of the structure of a real-time data synchronization task scheduling optimization system in an embodiment of this application.

[0096] It should be noted that, Figure 3 The structure of the real-time data synchronization task scheduling optimization system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0097] like Figure 3As shown, a real-time data synchronization task scheduling optimization system includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 302 or programs loaded from storage portion 308 into Random Access Memory (RAM) 303, such as executing the methods in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0098] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

Claims

1. A method for optimizing real-time data synchronization task scheduling, characterized in that, The method includes: Obtain the first foreign key dependency features of each data table in the source database, generate the first execution priority sequence of the synchronization task based on the first foreign key dependency features, and determine the first pre-execution state of the synchronization task. Extract the resource requirement identifier of the synchronization task, obtain the current resource quantity of the target node, and distribute the synchronization task to the target node according to the resource requirement identifier and the current resource quantity to obtain a task distribution plan; The synchronization task is executed on the target node according to the task distribution plan. When the operating load of the target node is greater than the preset load threshold, the unexecuted synchronization tasks on the target node are migrated to the backup node, and the task receiving status of the backup node is recorded. Obtain the operation change logs generated by the backup node and the target node during the execution of the synchronization task, and compare the execution time sequence of the operation change logs with the sorting result of the first execution priority sequence; When the execution time sequence is the same as the sorting result, a synchronization success confirmation signal is generated, and the synchronization completion flag of the source database is updated according to the synchronization success confirmation signal; When the first foreign key dependency feature changes, a second execution priority sequence is generated based on the changed second foreign key dependency feature, and a second preceding execution state for each synchronous task is determined.

2. The method according to claim 1, characterized in that, The step of generating a first execution priority sequence for the synchronization task based on the first foreign key dependency feature, and determining the first pre-execution state of the synchronization task, includes: Extract the primary table identifier and the secondary table identifier from the first foreign key dependency feature; The synchronization task corresponding to the master table identifier is determined as the pre-execution item, and the synchronization task corresponding to the slave table identifier is determined as the post-execution item; The execution priority sequence is obtained by arranging the preceding and following execution items according to their execution order; In the first execution priority sequence, obtain the synchronization task corresponding to the slave table identifier, and find the target synchronization task corresponding to each master table identifier associated with the synchronization task corresponding to the slave table identifier; Extract the execution completion status of the target synchronization task corresponding to each of the master table identifiers, and determine the execution completion status as the first preceding execution status of the synchronization task corresponding to the slave table identifier.

3. The method according to claim 1, characterized in that, The step of extracting the resource requirement identifier of the synchronization task, obtaining the current resource quantity of the target node, and distributing the synchronization task to the target node according to the resource requirement identifier and the current resource quantity to obtain a task distribution plan includes: Obtain the estimated data processing volume of the synchronization task, and use the estimated data processing volume as the resource requirement identifier; Obtain the remaining memory and processor idle rate of multiple candidate nodes, and determine the current resource quantity based on the remaining memory and processor idle rate; The candidate node whose resource requirement identifier is less than the current resource quantity is selected as the target node; The task distribution plan is generated based on the matching result between the synchronization task and the target node.

4. The method according to claim 1, characterized in that, The synchronization task is executed on the target node according to the task distribution plan. When the operating load of the target node exceeds a preset load threshold, the unexecuted synchronization tasks on the target node are migrated to a backup node, and the task reception status of the backup node is recorded, including: When the operating load of the target node exceeds a preset load threshold, a task pause command is sent to the target node; Obtain the task execution progress indicator when the target node receives the task pause command; Extract the unexecuted synchronization tasks based on the task execution progress identifier, and send the unexecuted synchronization tasks to the backup node; When the backup node returns a successful reception response, the successful reception response is recorded as the task reception status.

5. The method according to claim 4, characterized in that, After recording the successful reception response as the task reception status, the method further includes: Extract the standby node identifier from the task receiving status; Update the node routing information in the task distribution plan according to the backup node identifier; A task recovery command is sent to the backup node so that the backup node can continue to execute the unexecuted synchronization task based on the updated node routing information.

6. The method according to claim 1, characterized in that, The comparison of the execution time sequence of the operation change log with the sorting result of the first execution priority sequence includes: Extract the data table identifier and operation completion timestamp from the operation change log; The data table identifiers are arranged according to the operation completion timestamp to obtain the execution time sequence of the operation change log; According to the execution priority of each synchronous task in the first execution priority sequence, the corresponding data table identifiers are extracted sequentially to generate the sorting result; The execution time sequence of the operation change log is compared item by item with the data table identifier of the same position in the sorting result to obtain the comparison result.

7. The method according to claim 1, characterized in that, When the execution time sequence is the same as the sorting result, a synchronization success confirmation signal is generated, and the synchronization completion flag of the source database is updated according to the synchronization success confirmation signal, including: When the execution time sequence is the same as the sorting result, extract the transaction commit identifier from the operation change log; The transaction commit identifier is bound to the table name of the data table to generate the synchronization success confirmation signal; The synchronization success confirmation signal is sent to the control terminal of the source database, so that the control terminal modifies the status field of the data table to the synchronization completion flag.

8. The method according to claim 1, characterized in that, When the first foreign key dependency feature changes, a second execution priority sequence is generated based on the changed second foreign key dependency feature, and a second pre-execution state for each synchronization task is determined, including: Read the table structure definition file of the source database; Extract the current foreign key dependency features from the table structure definition file; Compare the current foreign key dependency feature with the first foreign key dependency feature; When the current foreign key dependency feature is different from the first foreign key dependency feature, the current foreign key dependency feature is used as the second foreign key dependency feature, and the second execution priority sequence is generated based on the second foreign key dependency feature.

9. The method according to claim 1, characterized in that, The step of obtaining the first foreign key dependency features of each data table in the source database includes: Send a metadata read request to the source database and receive the metadata set returned by the source database; Extract the constraint definition fields of each data table from the metadata set; For each of the data tables, extract the related fields from the constraint definition fields that correspond to the primary key of the related data table in the source database, and use the related fields as the first foreign key dependency feature of the data table.

10. A real-time data synchronization task scheduling optimization system, characterized in that, The real-time data synchronization task scheduling optimization system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the real-time data synchronization task scheduling optimization system to perform the method as described in any one of claims 1-9.

Images