Multi-database automatic synchronization method under AI platform

Abstract

The invention provides a multi-database automatic synchronization method under an AI platform, and belongs to the technical field of database automatic synchronization. The captured change records are converted into synchronous messages in a unified format, the synchronous messages are input into an intelligent scheduling optimization model for dependency analysis and transaction reordering based on dynamic programming, data conversion rules are dynamically generated according to mode version differences, and data consistency is verified through a Merkle tree hierarchical structure; the multi-stage self-adaptive compression transmission and data writing are completed after the target node executes conflict detection and circular dependency processing, and finally the final consistency among the multiple databases is guaranteed through a periodic full-amount account checking task, so that the technical problem of synchronization performance bottleneck caused by low transaction scheduling efficiency in the automatic synchronization process of the multiple databases is solved.

Description

Technical Field

[0001] This invention belongs to the field of database automatic synchronization technology, and more specifically, relates to a method for automatic synchronization of multiple databases under an AI platform. Background Technology

[0002] In distributed database systems and multi-datacenter architectures, automatic synchronization of multiple databases is a key technology for ensuring data consistency. Traditional database synchronization schemes typically employ a log-based change data capture mechanism combined with a simple first-in-first-out (FIFO) queue for transaction scheduling, executing captured data change operations sequentially during synchronization. However, this serial scheduling method based on fixed rules ignores the dependencies and resource contention characteristics between transactions. When the number of transactions to be synchronized increases and complex read-write dependencies exist, many transactions become stuck in a serial waiting state due to lock release, leading to a significant decrease in system throughput and a sharp increase in synchronization latency. In existing technologies, due to the lack of systematic analysis of inter-transaction dependencies and intelligent scheduling optimization mechanisms, multi-database synchronization systems cannot effectively identify sets of transactions that can be executed in parallel, nor can they dynamically adjust the execution order according to real-time load, resulting in a serious waste of computing resources and network bandwidth. In other words, existing technologies suffer from the technical problem of low transaction scheduling efficiency during automatic multi-database synchronization, leading to synchronization performance bottlenecks. Summary of the Invention

[0003] In view of this, the present invention provides a method for automatic synchronization of multiple databases under an AI platform, which can solve the technical problem of low transaction scheduling efficiency leading to synchronization performance bottleneck in the process of automatic synchronization of multiple databases in the prior art.

[0004] This invention is implemented as follows: An automatic multi-database synchronization method under an AI platform includes the following steps: initializing the multi-database synchronization topology, assigning a globally unique identifier to each database node, establishing a connection graph between nodes, recording the mode version number and data version vector of each database node, and starting a change data capture module; the change data capture module monitors the add, delete, and modify operations of each database node in real time, converting the captured change records into a unified format synchronization message; inputting the unified format synchronization message into an intelligent scheduling optimization model, which performs dependency analysis on the transactions to be synchronized and executes a transaction reordering optimization algorithm based on dynamic programming to generate an optimized transaction scheduling plan; dynamically generating data conversion rules based on the difference between the mode version number corresponding to the source node identifier and the mode version number corresponding to the target node in the optimized transaction scheduling plan; dividing the synchronized data after conversion into multiple data blocks according to the primary key range, calculating the hash value for each data block to construct a Merkle tree hierarchy; starting an improved Gossip protocol propagation module; performing conflict detection when the target database node receives synchronized data; decompressing the received compressed synchronized data, executing the optimized transaction scheduling plan to complete data writing and update the local data version vector, and performing a periodic full reconciliation task.

[0005] The globally unique identifier is a 128-bit binary identifier consisting of the node deployment geographic location code, the node type code, and an incrementing sequence number.

[0006] The data version vector is an ordered array that records the latest operation version number of each database node. The length of the ordered array is equal to the total number of nodes in the multi-database synchronization topology.

[0007] Specifically, the captured change records are converted into a unified format synchronization message. This involves extracting the operation type, target table name, primary key value, data before the change, and data after the change from the change records, and encapsulating them into a unified format synchronization message structure containing a message header and a message body.

[0008] The structure of the intelligent scheduling optimization model is as follows: the input layer receives the read and write set information and timestamp information of the set of transactions to be synchronized; the encoding layer extracts the dependency relationship features between transactions through a multi-head attention mechanism; the feature extraction layer uses a graph convolutional neural network to learn node features of the directed graph constructed by the dependency relationship; the dynamic programming optimization layer executes a transaction reordering optimization algorithm based on dynamic programming based on the learned feature vectors; and the output layer generates an optimized transaction scheduling plan that includes the transaction execution order and parallel batch partitioning.

[0009] Among them, the transaction reordering optimization algorithm based on dynamic programming specifically analyzes the read set and write set of each transaction in the set of transactions to be synchronized to identify data dependencies, constructs a directed acyclic dependency graph, defines the dynamic programming state as the combination of the set of scheduled transactions and the current time point, selects all transactions whose predecessor dependencies have been satisfied as candidate transactions from the unscheduled transactions during state transition, and recursively executes state transitions until all transactions are scheduled.

[0010] Specifically, data conversion rules are dynamically generated based on the difference between the mode version number corresponding to the source node identifier and the mode version number corresponding to the target node in the optimized transaction scheduling plan. This involves querying the current mode version numbers of the source database and the target database, retrieving all mode change records between the two mode version numbers from the mode management system, and generating data conversion rules based on the mode change records.

[0011] Specifically, the Merkle tree hierarchy is constructed by dividing the entire table into several data blocks based on the primary key range. All records within each data block are sorted according to the primary key range and then concatenated into a byte stream. The SHA-256 hash value is calculated as the hash of the leaf node. The hash values ​​of two adjacent leaf nodes are concatenated and the SHA-256 hash value is calculated again as the hash of the parent node. The concatenation and hash calculation process is repeated until a unique root hash value is generated.

[0012] Specifically, data consistency is determined and discrepancy data blocks are located by comparing the root hash values ​​of the source database and the target database. If the root hash values ​​are different, the hash values ​​of the left and right child nodes of the root node are compared. The process recursively traverses the subtree branches with different hash values ​​until the leaf node hash is reached, and records the primary key range of the data blocks corresponding to the hash values ​​of all leaf nodes with different hash values.

[0013] Specifically, the steps of the improved Gossip protocol propagation module are as follows: each database node maintains a list of neighboring nodes and a local data version vector; the database node randomly selects a fixed number of neighboring nodes from the list of neighboring nodes at set time intervals and sends a message containing the local data version vector and a summary of the latest change log to the selected neighboring nodes.

[0014] Specifically, when the target database node receives synchronized data, it performs conflict detection. Specifically, it resolves data conflicts based on global timestamps and business priority rules, constructs a global waiting directed graph for detected circular dependencies, selects the transaction with the least cost for rollback, and performs multi-level adaptive compression processing on the synchronized data to be transmitted before transmitting it over the network.

[0015] Specifically, the circular dependency detection and handling involves each database node periodically collecting lock wait information from local transactions, sending the lock wait information to a distributed deadlock detection service to construct a global wait directed graph, performing topological sorting on the global wait directed graph to detect loops, and selecting the transaction with the lowest cost from the detected loops for rollback.

[0016] The multi-level adaptive compression process specifically analyzes the data type distribution of the synchronized data. For data blocks where text type fields account for more than 60%, a dictionary encoding method is used to build a high-frequency vocabulary, and the original text is replaced with an index before run-length encoding is performed on the index sequence. For data blocks where numeric type fields account for more than 60%, the difference between adjacent numeric values ​​is calculated and the difference is represented by variable-length integer encoding.

[0017] The global timestamp is generated using a hybrid logic clock, which combines a physical clock and a logic counter. When the physical clocks are the same, the logic counter is used to distinguish the order of events.

[0018] The mode management system assigns an incrementing global mode version number for each mode change. The global mode version number is represented by a 64-bit integer, with the high 32 bits being the timestamp in seconds of the change and the low 32 bits being the incrementing sequence number within the same second.

[0019] In variable-length integer encoding, the highest bit of each byte is used as a continuation flag. When the highest bit is 1, it indicates that there are more bytes to follow. When the highest bit is 0, it indicates that the current byte is the last byte. The remaining 7 bits store the actual value.

[0020] This invention constructs an intelligent scheduling optimization model and combines it with a transaction reordering optimization algorithm based on dynamic programming. This achieves intelligent optimization of the dependency analysis and execution order of transactions to be synchronized, solving the performance bottleneck problem caused by traditional serial scheduling. The intelligent scheduling optimization model uses a multi-head attention mechanism and a graph convolutional neural network to extract complex dependency features between transactions. Through a dynamic programming algorithm, it reorganizes the originally serially waiting transactions into multiple parallel batches while ensuring that dependency constraints are satisfied. This significantly reduces idle waiting time caused by lock contention. Simultaneously, the state transition process comprehensively considers a multi-dimensional cost function that integrates execution time, lock conflict probability, and resource consumption, resulting in a better balance between throughput and latency in the generated scheduling plan. Compared to traditional rule-based fixed scheduling strategies, the intelligent scheduling mechanism of this invention can learn implicit scheduling patterns based on historical experience and adapt to dynamically changing workloads. In summary, this invention solves the technical problem mentioned in the background art where low transaction scheduling efficiency leads to synchronization performance bottlenecks during automatic synchronization of multiple databases. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method of the present invention.

[0022] Figure 2 This is a flowchart of the change data capture and message encapsulation process in the embodiment.

[0023] Figure 3 This is a diagram of the Merkle tree construction and differential localization structure in the embodiment.

[0024] Figure 4 This is a flowchart of the multi-level adaptive compression process in the embodiment.

[0025] Figure 5 This is a statistical chart of system performance monitoring data in the embodiment. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below.

[0027] like Figure 1 As shown, the invention provides a method for automatic synchronization of multiple databases under an AI platform, including:

[0028] S1. Initialize the multi-database synchronization topology, assign a globally unique identifier to each database node, establish a connection graph between nodes, record the schema version number and data version vector of each database node, and start the change data capture module.

[0029] S2. The change data capture module monitors the add, delete, and modify operations of each database node in real time, and converts the captured change records into a unified format synchronization message. The unified format synchronization message includes the source node identifier and a global timestamp.

[0030] S3. Input the unified format synchronization message into the intelligent scheduling optimization model. The intelligent scheduling optimization model performs dependency analysis on the transaction to be synchronized and executes a transaction reordering optimization algorithm based on dynamic programming to generate an optimized transaction scheduling plan. Based on the difference between the mode version number corresponding to the source node identifier and the mode version number corresponding to the target node in the optimized transaction scheduling plan, data conversion rules are dynamically generated.

[0031] S4. Divide the synchronized data after transformation by the data transformation rules into multiple data blocks according to the primary key range. Calculate the hash value of each data block to construct a Merkle tree hierarchy. Determine data consistency and locate the discrepancy data blocks by comparing the root hash values ​​of the source database and the target database.

[0032] S5. Start the improved Gossip protocol propagation module. Each database node periodically and randomly selects neighboring nodes to exchange change log summaries and the data version vector. The missing change records are identified by comparing the data version vectors and pushed to the target database node.

[0033] S6. When the target database node receives synchronization data, it performs conflict detection, resolves data conflicts according to the global timestamp and business priority rules, constructs a global waiting directed graph for the detected circular dependencies and selects the transaction with the least cost for rollback, and performs multi-level adaptive compression processing on the synchronization data to be transmitted before transmitting it through the network.

[0034] S7. The target database node decompresses the received compressed synchronization data, executes the optimized transaction scheduling plan to complete the data writing and update the local data version vector, and executes a periodic full reconciliation task to compare the data integrity of each node using the Merkle tree hierarchy structure and trigger the repair synchronization process.

[0035] The globally unique identifier is a 128-bit binary identifier consisting of a node deployment geographic location code, a node type code, and an incrementing sequence number.

[0036] The data version vector is an ordered array that records the latest operation version number of each database node. The length of the ordered array is equal to the total number of nodes in the multi-database synchronization topology, and each element of the ordered array corresponds to the version number of a database node.

[0037] The change data capture module deploys log parsers on each database node to monitor add, delete, and modify operations in real time.

[0038] The specific steps of converting the change record into a unified format synchronization message include: extracting the operation type, target table name, primary key value, data before the change, and data after the change from the change record; encapsulating the extracted operation type, target table name, primary key value, data before the change, and data after the change into a unified format synchronization message structure containing a message header and a message body, wherein the message header contains the source node identifier, global timestamp, message sequence number, and message type, and the message body contains the data before the change and the data after the change; and encoding the data fields in the message body according to the character set of the target database.

[0039] The specific structure of the intelligent scheduling optimization model is as follows: the input layer receives the read / write set information and timestamp information of the set of transactions to be synchronized; the encoding layer extracts the dependency relationship features between transactions through a multi-head attention mechanism; the feature extraction layer uses a graph convolutional neural network to learn node features on the directed graph constructed by the dependency relationship; the dynamic programming optimization layer executes the transaction reordering optimization algorithm based on the learned feature vectors; and the output layer generates an optimized transaction scheduling plan that includes the transaction execution order and parallel batch partitioning. The steps for establishing the training dataset of the intelligent scheduling optimization model specifically include: collecting transaction execution records from historical synchronization tasks, recording the read / write set, execution time, number of lock conflicts, and final execution order for each transaction; and labeling the data dependencies between transactions, marking read-after-write dependencies, write-after-write dependencies, and so on. The model identifies read dependencies and write-after-write dependencies; it calculates the total execution time and resource utilization under different scheduling orders as optimization target labels; it divides the collected transaction execution records into training and validation sets in an 8:2 ratio; the training steps of the intelligent scheduling optimization model specifically include: initializing model parameters and setting the learning rate to 0.001 and the batch size to 64; inputting the set of transactions to be synchronized in the training set into the intelligent scheduling optimization model, which outputs a predicted scheduling plan; comparing the predicted scheduling plan with the actual optimal scheduling plan, calculating the scheduling quality loss function, which comprehensively considers execution time deviation and dependency constraint violation penalties; updating the model parameters using the backpropagation algorithm, and iteratively training until the scheduling quality loss function on the validation set converges.

[0040] The specific steps of the transaction reordering optimization algorithm based on dynamic programming include: analyzing the read and write sets of each transaction in the set of transactions to be synchronized; identifying data dependencies by comparing the read and write sets of different transactions; marking read-write dependency edges from A to B when the write set of transaction A intersects with the read set of transaction B; constructing a directed acyclic dependency graph based on the identified data dependencies, where nodes represent transactions and directed edges represent dependency constraints; defining the dynamic programming state as a combination of the set of scheduled transactions and the current time point; selecting all transactions whose predecessor dependencies are satisfied from the unscheduled transactions as candidate transactions during state transition; and calculating the incremental time to add each candidate transaction to the current schedule. The incremental cost is determined by comprehensively considering the execution time of the candidate transaction, the probability of lock conflicts with the already scheduled transaction set, and resource consumption. The state transition equation selects the candidate transaction with the smallest incremental cost in the current dynamic programming state and adds it to the scheduling sequence, while simultaneously updating the current time point to the expected completion time of the candidate transaction. All transactions with zero in-degree and no data dependencies in the directed acyclic dependency graph are identified as forming parallel batches, and transactions within these parallel batches are marked as allowed for concurrent execution in the optimized transaction scheduling plan. The state transition is recursively executed until all transactions are scheduled, and the optimal decision path in the dynamic programming state process is backtracked to generate the final transaction execution order and the parallel batch partitioning scheme.

[0041] The transaction reordering optimization algorithm based on dynamic programming provides the following technical effects to the intelligent scheduling optimization model: By systematically analyzing the data dependencies and resource contention patterns between transactions, the algorithm maximizes parallel execution opportunities while ensuring the correctness of transaction execution. It reorganizes transactions that originally required sequential waiting into multiple parallel batches, significantly reducing idle waiting time caused by lock contention. The dynamic programming method avoids repeated computation of the same subproblems through memoization search, solving an approximately optimal scheduling scheme in polynomial time. The state transition process comprehensively considers a multi-dimensional cost function of execution time, lock contention, and resource consumption, resulting in a well-balanced optimized transaction scheduling plan between throughput and latency. For complex... For large-scale sets of transactions to be synchronized with complex dependencies, the dynamic programming-based transaction reordering optimization algorithm ensures the satisfaction of dependency constraints through topological sorting. Simultaneously, it utilizes a greedy strategy to select a locally optimal solution at each decision point, resulting in overall scheduling quality approaching global optimality. By combining the dynamic programming-based transaction reordering optimization algorithm with neural network feature extraction capabilities, the intelligent scheduling optimization model can learn implicit patterns from historical scheduling experience. It can quickly predict potential conflict points for new sets of transactions to be synchronized and adjust the transaction execution order in advance. Compared to traditional rule-based schedulers, the intelligent scheduling optimization model exhibits stronger adaptability to dynamically changing workloads and can automatically adjust scheduling strategies based on real-time system status monitoring, avoiding performance degradation issues of fixed strategies in different scenarios.

[0042] The specific steps for dynamically generating data conversion rules based on the difference between the mode version number and the mode version number include: querying the current mode version number of the source database and the target database; retrieving all mode change records between the two mode version numbers from the mode management system, wherein the mode change record includes the change type, the changed field name, the field data type, and the constraints; for changes involving newly added field types, inserting field default value filling logic into the data conversion rule, wherein the field default value is selected as zero, an empty string, or a null value according to the field data type; for changes involving deleted field types, inserting field filtering logic into the data conversion rule to prevent the data of the corresponding field in the unified format synchronization message from being written to the target database; for mode change records of the type change type, inserting type conversion logic into the data conversion rule, wherein the type conversion logic includes precision adjustment for numeric types, length truncation for character types, and format conversion for time types; and organizing all generated data conversion rules into a conversion rule mapping table according to the changed field name.

[0043] The specific steps for constructing the Merkle tree hierarchy include: dividing the entire table data into several data blocks according to the primary key range, with each data block containing all records within a continuous primary key range; sorting all records within each data block according to the primary key range and concatenating them into a byte stream, calculating the SHA-256 hash value of the byte stream as the leaf node hash; concatenating the hash values ​​of two adjacent leaf nodes and calculating the SHA-256 hash value again as the parent node hash, repeating the concatenation and hash calculation process until a unique root hash value is generated; and recording the Merkle tree hierarchy information, which includes the hash value of each node, the parent node pointer, and the child node pointer.

[0044] The steps for determining data consistency and locating differing data blocks using the root hash value specifically include: obtaining the root hash values ​​of the corresponding data tables in the source and target databases; comparing whether the two root hash values ​​are the same, and if they are the same, determining that the data is completely consistent and no synchronization is needed; if the root hash values ​​are different, continuing to compare the hash values ​​of the left and right child nodes of the root node to locate the subtree branches with different hash values; recursively traversing the subtree branches with different hash values ​​downwards, comparing the child node hash values ​​layer by layer until the leaf node hash is reached; recording the primary key range of the data blocks corresponding to the hash values ​​of all leaf nodes with different hash values, and the primary key range of the data blocks identifying the differing data blocks.

[0045] The steps performed by the improved Gossip protocol propagation module specifically include: each database node maintaining a list of neighboring nodes and a local data version vector, wherein the local data version vector records the latest operating version number of each node known to the node; the database node randomly selects a fixed number of neighboring nodes from the list of neighboring nodes at set time intervals; sends a message containing the local data version vector and a summary of the most recent change log to the selected neighboring nodes; receives the remote data version vector returned by the neighboring nodes, and identifies the node identifier in the remote data version vector that is greater than the corresponding element in the local data version vector by comparing elements one by one; requests the missing change record corresponding to the node identifier from the neighboring nodes, receives and merges the missing change record into the local database; and updates the version number of the corresponding node in the local data version vector to the received latest version number.

[0046] The change log summary includes the sequence number of the change record, the operation type, and the timestamp.

[0047] The specific steps for detecting and handling circular dependencies include: each database node periodically collecting lock wait information for local transactions, the lock wait information including the transaction identifier, the identifier of the lock being waited for, and the identifier of the transaction holding the lock; sending the lock wait information to a distributed deadlock detection service, the distributed deadlock detection service receiving the lock wait information from all database nodes and constructing a global wait directed graph; performing a topological sort on the global wait directed graph, if a node that cannot continue is detected during the topological sort, a cycle is determined to exist, i.e., a circular dependency; selecting the transaction with the lowest cost from the detected cycle and rolling it back, the cost of the transaction with the lowest cost being calculated based on the transaction's execution time and the amount of data modified; sending a rollback instruction to the database node holding the rolled-back transaction, the database node executing the transaction rollback and releasing the occupied lock resources.

[0048] The multi-level adaptive compression process specifically includes: analyzing the data type distribution of the synchronized data and statistically analyzing the proportions of text, numeric, and binary fields; for data blocks where text fields account for more than 60%, establishing a high-frequency vocabulary using dictionary encoding and replacing the original text with an index, then performing run-length encoding on the index sequence to compress consecutive repeating values; for data blocks where numeric fields account for more than 60%, calculating the difference between adjacent values ​​and representing the difference using variable-length integer encoding, which dynamically allocates the number of bytes based on the value size; skipping compression processing and directly transmitting the already compressed binary fields; and dynamically adjusting the compression level based on the current system CPU utilization: reducing the compression level to reduce computational overhead when the system CPU utilization exceeds 80%, and increasing the compression level to increase the compression ratio when the system CPU utilization is below 40%.

[0049] The periodic full reconciliation task is executed every 24 hours, and the repair synchronization process re-executes the change data capture, conflict detection and data writing steps for the difference data blocks.

[0050] Furthermore, the mode management system assigns an incrementing global mode version number for each mode change. The global mode version number is represented by a 64-bit integer, with the high 32 bits being the timestamp in seconds of the change and the low 32 bits being the incrementing sequence number within the same second.

[0051] Furthermore, the global timestamp is generated using a hybrid logic clock, which combines a physical clock and a logic counter. When the physical clocks are the same, the logic counter is used to distinguish the order of events.

[0052] Furthermore, the data conflict resolution rule prioritizes the comparison of the global timestamps. When the global timestamps are the same, the lexicographical order of the source node identifiers is used for comparison, and the data of the database node with the larger lexicographical order is taken as the final value.

[0053] Furthermore, the business priority rule assigns priority weight values ​​to different data tables based on the business importance and data freshness requirements of the data tables, and the priority weight values ​​range from 1 to 10.

[0054] Furthermore, the size of the high-frequency vocabulary in the dictionary encoding is limited to 4096 entries, and the minimum repetition threshold for the run-length encoding is 3.

[0055] Furthermore, the variable-length integer encoding uses the highest bit of each byte as a continuation flag. When the highest bit is 1, it indicates that there are more bytes to follow. When the highest bit is 0, it indicates that the current byte is the last byte, and the remaining 7 bits store the actual value.

[0056] Furthermore, the set time interval is 100 milliseconds to 500 milliseconds, and the fixed number of neighboring nodes is 3 to 5.

[0057] The present invention also provides a method for forming an automatic synchronization system of multiple databases under an AI platform by means of a computer, wherein the computer is provided with a readable storage medium, the readable storage medium stores program instructions, and the program instructions execute the above-described method when running in the computer.

[0058] The specific implementation methods of the above steps are described in detail below.

[0059] The specific implementation of step S1 is as follows: First, scan all database nodes to be connected and collect their network addresses and database type information. Generate an 8-bit region code based on the geographical location of the node deployment, and assign a 4-bit type code based on the database type, such as relational or non-relational. Combine the current millisecond timestamp and auto-incrementing sequence number to construct a 128-bit globally unique identifier. This identifier is stored in binary format to save space and support fast comparison. Then, establish logical connection relationships between nodes through configuration files or service discovery mechanisms and store the topology in the form of an adjacency list. Initialize the mode version number of each node to an initial value of 1 and create a data version vector with a length equal to the total number of nodes. Each element in the vector is initialized to 0 to indicate that no operation record has been received from the corresponding node. At the same time, deploy a change data capture module on each node. This module realizes real-time monitoring of add, delete, and modify operations by parsing database transaction logs or trigger mechanisms. The initialization process establishes a complete synchronization infrastructure to provide topology basis for subsequent data flow.

[0060] The specific implementation of step S2 is as follows: The change data capture module continuously reads the database's write-ahead log or binary log file, identifies INSERT log entries representing insert operations, UPDATE log entries representing update operations, and DELETE log entries representing delete operations through the log parser, extracts the operation type identifier, the name of the target table being operated on, the primary key field of the record and its value from the log, and for update operations, extracts both the old value before the change and the new value after the change. The extracted raw data is encapsulated according to a predefined message structure, the message header is filled with the globally unique identifier of the source node to indicate the source of the change, a global timestamp generated by a hybrid logical clock is embedded for subsequent conflict resolution, an incremental message sequence number is assigned to ensure message ordering, and the message type is labeled, such as incremental synchronization or full synchronization. The message body stores the complete data record before and after the change, and the character type fields in the message body are transcoded according to the character set encoding standard of the target database to avoid garbled characters. The generated unified format synchronization message is pushed to the message queue to await further processing. This step, through standardized message format, shields the differences between heterogeneous databases and lays the foundation for cross-platform synchronization.

[0061] The specific implementation of step S3 involves batch reading the set of transactions to be synchronized from the message queue and extracting the read and write set information involved in each transaction. This transaction information is then input into the input layer of the intelligent scheduling optimization model. This input layer represents the read and write sets of transactions as vectors and adds a global timestamp as a temporal feature. The encoding layer uses a multi-head attention mechanism to calculate the correlation matrix between transactions and identifies transaction pairs with data competition through attention weights. The feature extraction layer constructs a directed graph of dependencies between transactions and uses a graph convolutional neural network to propagate node features. It learns the global dependency pattern by aggregating neighbor node information through multi-layer graph convolution. The dynamic programming optimization layer receives the feature vector output by the graph convolution and executes a transaction reordering algorithm. This algorithm first analyzes the intersection relationship of the transaction read and write sets, marking three types of constraint edges: read-after-write dependency, write-after-read dependency, and write-after-write dependency. After constructing the dependency graph, the state space is defined as the combination of the scheduled transaction set and the current time step. During state transition, all candidate transactions whose predecessor dependencies are satisfied are enumerated, and their incremental costs are calculated. The cost function comprehensively considers the transaction execution time, the probability of lock conflicts with scheduled transactions obtained through historical statistics, and memory and CPU resource usage. It selects the transaction with the lowest cost to add to the scheduling sequence and updates the current time to the transaction's expected completion time. At the same time, it identifies transactions without dependencies to form parallel batches to improve throughput. It recursively executes state transitions until all transactions are scheduled and backtracks the decision path to generate the optimal scheduling plan. The output layer queries the mode management system to obtain all intermediate mode change records based on the difference in mode version numbers between the source node and the target node in the scheduling plan. For newly added fields, it inserts default values ​​selected according to the data type into the conversion rules, such as filling numeric types with 0 and character types with empty strings. For deleted fields, it inserts filtering logic to prevent the corresponding data from being written to the target database. For type changes, it inserts conversion logic, such as preserving precision or truncating when converting integers to floating-point types. All conversion rules are organized into a mapping table by field name for subsequent data processing. This step automatically discovers the optimal scheduling strategy through a deep learning model, avoiding the limitations of manual parameter tuning.

[0062] The specific implementation of step S4 is as follows: Based on the value range of the primary key in the data table, the entire table is divided into several consecutive data blocks. Each block contains all rows within a fixed primary key range. Records within each data block are sorted in ascending order by primary key and then serialized into a byte stream. The SHA-256 hash value of this byte stream is calculated as the hash identifier of the leaf node. The SHA-256 algorithm is collision-resistant, ensuring that different data contents generate different hash values. The hash values ​​of two adjacent leaf nodes are concatenated in a fixed order, and the SHA-256 hash is calculated again to obtain the hash value of the parent node at the next higher level. This concatenation and hash calculation process is repeated, merging upwards layer by layer until a unique hash value is generated for the root node of the spanning tree. The system maintains the hierarchical structure information of the tree, including the hash values ​​of each node and their parent-child pointer relationships to facilitate subsequent traversal. It obtains the root hash values ​​of the corresponding tables in the source and target databases and compares them. If they are exactly the same, it means that the data at both ends is consistent and no synchronization operation is required. If the root hash values ​​are different, it continues to compare the hash values ​​of the left and right child nodes of the root node to locate the subtree branch where the difference is located. It recursively traverses the branches with mismatched hash values ​​downwards and compares the hash values ​​of child nodes layer by layer until it reaches the leaf node. It records the primary key range of the data blocks corresponding to all leaf nodes with different hash values ​​as the difference data blocks that need to be synchronized. This step utilizes the incremental verification feature of Merkle trees to greatly reduce the amount of data transmission for full comparison and improve verification efficiency.

[0063] The specific implementation of step S5 is as follows: Each database node maintains a neighbor list containing neighbor node identifiers and network addresses, as well as a local data version vector recording the latest operation version number of each node. Nodes periodically perform a propagation task at time intervals of 100 milliseconds to 500 milliseconds. Each time, a pseudo-random number generator randomly selects 3 to 5 neighbor nodes from the neighbor list as the communication targets for this round. A propagation message containing the local data version vector and a summary of the change log for the recent period is sent to the selected neighbors. The change log summary includes the sequence number of the change record, the operation type identifier, and timestamp information so that the receiver can quickly determine whether a complete record is needed. The remote data version vector returned by the neighbor is then received. After measuring, the local and remote vectors are compared element by element. When the version number of a certain position in the remote vector is greater than the corresponding position in the local vector, it indicates that the local vector is missing some changes for that node. The missing change record is requested from the neighboring node and the complete change data is received. The received change records are merged into the local database in timestamp order and the corresponding add, delete and modify operations are performed. The version number of the corresponding position in the local data version vector is updated to the latest version number received to reflect the known local state. This step uses an improved Gossip protocol to achieve eventual consistency through randomized node selection and incremental propagation mechanism. Even if some nodes are temporarily unreachable, the change information can eventually reach all nodes through multi-hop propagation.

[0064] The specific implementation of step S6 is as follows: When the target database node receives synchronization data from the network, it first extracts the global timestamp in the message header and compares it with the timestamp of the existing local record. If the timestamp of the new data is larger, it is determined that the new data is a newer version. If the timestamps are the same, the lexicographical order of the source node identifiers is further compared. The node data with the larger lexicographical order is used as the final retained value to resolve conflicts in a deterministic manner. For dependencies involving multiple transactions, the target node collects the lock wait information of local transactions, including the transaction identifier, the identifier of the lock object being waited for, and the identifier of the transaction currently holding the lock. The lock wait information is reported to the distributed deadlock detection service. This service aggregates the lock wait information of all nodes to construct a global wait directed graph. In the graph, nodes represent transactions and edges represent wait relationships. A topological sorting algorithm is executed on the directed graph to attempt to generate a linear sequence. If a set of nodes with a non-zero in-degree is found during the sorting process, it is determined that a cycle exists, i.e., a circular dependency or a deadlock. The transaction with the lowest cost is selected from the detected cycle and rolled back. The cost is calculated based on the execution time of the transaction and the amount of data modified. A rollback instruction is sent to the node holding the selected transaction and the relevant lock resources are released to break the deadlock. For the synchronization data to be transmitted... The data is first analyzed to determine the distribution of text, numeric, and binary fields. If text fields account for more than 60%, a dictionary encoding method is used to scan the data and build a dictionary containing 4096 high-frequency words. After replacing the original text with index numbers, run-length encoding is performed on the index sequence. Run-length encoding compresses the same index that appears at least 3 times consecutively into a combination of a count value and an index value. If numeric fields account for more than 60%, the difference between adjacent values ​​is calculated and represented by variable-length integer encoding. This encoding uses the highest bit of each byte as a continuation flag. When the highest bit is 1, it indicates that there are more bytes to follow; when the highest bit is 0, it indicates the end. The remaining 7 bits store the actual value. For compressed binary fields, such as images or encrypted data, they are transmitted directly without compression to avoid negative optimization. The compression level is dynamically adjusted according to the system CPU utilization. When the CPU utilization exceeds 80%, the compression level is reduced to reduce computational overhead; when the CPU utilization is below 40%, the compression level is increased to increase the compression ratio. The compressed data is transmitted to the target node through the network. This step optimizes network bandwidth utilization while ensuring data consistency through multi-dimensional conflict detection and adaptive compression strategies.

[0065] The specific implementation of step S7 is as follows: After receiving the compressed data, the target database node performs the corresponding decompression operation according to the compression type marked in the message header. For dictionary-encoded data, the dictionary table is read first, and then the index sequence is restored to the original text. For run-length encoded data, the count value is expanded to restore the continuously repeating index. For variable-length integer encoded data, it is read byte by byte, and the continuation flag is used to determine whether the next byte needs to be read. The lower 7 bits of each byte are concatenated to restore the complete value. After decompression, the data writing operation is performed in the order and parallel batches specified in the optimized transaction scheduling plan. For transaction batches marked as parallel execution, multi-threading or multi-process writing is started simultaneously to improve throughput. For transactions with dependencies, the scheduling is strictly followed. The sequential execution ensures correctness. After each transaction's write operation is completed, the version number of the corresponding source node in the local data version vector is updated to the version number carried by that transaction. The system periodically starts a full reconciliation task every 24 hours. This task uses the Merkle tree hierarchy structure constructed in step S4 to compare the data integrity between nodes. It quickly locates data blocks with differences by comparing the root hash value. For data blocks with differences, a repair synchronization process is triggered. The repair process is re-executed to capture the latest data of the difference blocks, and after conflict detection, it is written to the target node to complete the data repair. This step, through the periodic reconciliation mechanism, promptly discovers and repairs data inconsistencies caused by network partitions or node failures to ensure the system's eventual consistency.

[0066] It should be noted that this invention also solves the following technical problem: in a multi-database heterogeneous environment, inconsistent schema version evolution of different database nodes leads to incompatible synchronization data formats. This invention records a schema version number for each database node and maintains a complete schema change history in the schema management system. During synchronization, it dynamically generates data conversion rules based on the differences in schema version numbers between the source and target nodes. For newly added fields, it inserts default value filling logic; for deleted fields, it inserts filtering logic; and for type changes, it inserts precision adjustment and format conversion logic. All conversion rules are organized into a mapping table for the synchronization engine to call, thereby achieving automated data conversion across schema versions. This ensures that synchronization operations can still be executed correctly even in scenarios with frequent database schema evolution, avoiding synchronization failures or data corruption caused by schema mismatch.

[0067] This invention addresses the technical problem of excessively long reconciliation times and high system resource consumption caused by traditional full-data reconciliation methods that require row-by-row comparison in large-scale distributed database clusters. By constructing a Merkle tree hierarchical structure, the data table is divided into multiple data blocks according to the primary key range, and a hash value is calculated for each data block. A unique root hash value is generated layer by layer from bottom to top. During reconciliation, the root hash values ​​of the source and destination are first compared to determine overall consistency. When a difference is detected, the subtree branches with different hash values ​​are recursively traversed downwards to locate the specific difference data block. Compared to the row-by-row full-data-scan method, this hierarchical verification mechanism reduces the time complexity of reconciliation from linear to logarithmic, significantly reducing the amount of data that needs to be transmitted and compared. Furthermore, the Merkle tree structure supports incremental updates; when data changes, only the hash values ​​on the affected paths need to be recalculated without rebuilding the entire tree, significantly reducing the computational overhead and network bandwidth consumption of periodic reconciliation tasks.

[0068] Specifically, the principle of this invention is as follows: This invention can solve the technical problem of low transaction scheduling efficiency in automatic synchronization of multiple databases. Its fundamental principle lies in transforming the traditional static rule scheduling into an intelligent dynamic scheduling mechanism based on dependency analysis. First, by systematically analyzing the read and write sets of each transaction to be synchronized, three types of data dependencies—read-after-write, write-after-read, and write-after-write—are accurately identified, and a directed acyclic dependency graph is constructed to express these constraints, providing a theoretical basis for subsequent parallel execution. Second, the transaction reordering algorithm based on dynamic programming defines the combined state of the already scheduled transaction set and the current time point. During state transition, it selects all transactions whose predecessor dependencies are satisfied as candidate sets and selects the optimal transaction to add to the scheduling sequence based on the incremental cost. This memoized search method avoids redundant calculations and solves for an approximate optimal solution in polynomial time. Third, the intelligent scheduling optimization model learns implicit patterns in historical scheduling experience through neural networks, enabling it to predict potential conflict points and adjust the transaction order in advance. Compared with manually designed heuristic rules, it has stronger generalization ability and adaptability. Finally, by identifying transactions with zero in-degree in the dependency graph to form parallel batches, the concurrent execution capability of the database system is fully utilized, transforming transactions that were originally waiting serially due to conservative scheduling into parallel execution, thereby improving the overall synchronization throughput and reducing synchronization latency.

[0069] The following provides a specific embodiment 1 of the present invention. The specific implementation methods of steps S1 and S2 in this embodiment 1 are the same as those described above, and will not be repeated in detail here. The specific implementation methods of other steps are described in detail below.

[0070] The specific implementation of step S3 involves batch reading the set of transactions to be synchronized from the message queue and extracting the read / write set information involved in each transaction. This transaction information is then input into an intelligent scheduling optimization model for processing. This model extracts the dependency features between transactions through a multi-head attention mechanism and a graph convolutional neural network, and then executes a transaction reordering algorithm at the dynamic programming optimization layer. The state transition equation of this algorithm is expressed as follows:

[0071] ;

[0072] In the formula, Set of scheduled transactions At the present moment The optimal scheduling cost is dimensionless. The set of candidate transactions whose predecessor dependencies have been satisfied; Number the candidate transactions; For matters The execution time, in seconds, is obtained through historical statistics; The maximum execution time among all pending transactions, in seconds, is obtained by taking the maximum execution time of all transactions in the set of pending transactions. The current time is expressed in seconds. For matters With the set of scheduled transactions The lock conflict probability is dimensionless and is calculated statistically from historical lock waiting records. The maximum lock collision probability is dimensionless and has an empirical value of 1. For matters Resource usage, in megabytes, is obtained through database query plan analysis; The maximum resource usage, in megabytes, is obtained by taking the maximum value of the resource usage of all transactions in the set of transactions to be scheduled. These are weighting coefficients, dimensionless, with default values ​​of 0.4, 0.3, and 0.3. Indicates that the transaction Add to the scheduled set The new set afterwards; This is a function that takes the minimum value.

[0073] Among them, lock conflict probability The calculation formula is expressed as follows:

[0074] ;

[0075] In the formula, For matters With transactions The proportion of the intersection of the read and write sets is dimensionless and calculated as follows: ,in For matters The write set, containing transactions The identifiers of all modified data items are obtained through parsing the transaction log. For matters The read set, containing transactions The identifiers of all data items read are obtained through parsing the transaction log. Indicates the number of elements in the set. This represents the intersection operation of sets; For matters and affairs The time window overlap factor is dimensionless. It takes a value of 1 when the expected execution times of two transactions overlap, and a value of 0 when they do not overlap. The determination method is as follows: and but ,otherwise ,in For matters The start time, in seconds. For matters The start time, in seconds. For matters Execution time, in seconds; Set of scheduled transactions Transaction number in the data; This is the summation operator.

[0076] When dynamically generating data conversion rules based on schema version number differences, the default value filling rule for schema changes involving newly added field types is described as follows:

[0077] ;

[0078] In the formula, The populated value is for the newly added field; the field type is obtained through the data type metadata in the schema management system, which is recorded in the field data type attribute of the schema change record.

[0079] The formula for adjusting numerical precision, based on the conversion rules for type changes, is as follows:

[0080] ;

[0081] In the formula, For the target database field value; The source database field values ​​are extracted from the message body of the unified format synchronization message. The precision of the target field is dimensionless and is obtained through the decimal places attribute defined in the target database schema. This is a rounding function that rounds the input value to the nearest integer.

[0082] The specific implementation of step S4 involves dividing the entire table into several data blocks based on the value range of the primary key, and constructing a Merkle tree hierarchical structure for each data block. The formula for calculating the hash value of the leaf nodes is as follows:

[0083] ;

[0084] In the formula, For the first The leaf node hash value of each data block is a 256-bit binary sequence; For the first The first data block The byte stream representation of each record is obtained by serializing it after sorting by the primary key. The serialization method is to convert the values ​​of each field of the record into byte arrays in the field order and then concatenate them. For the first The total number of records contained in each data block is obtained by counting the number of records within the primary key range; This is a byte stream concatenation function that joins multiple byte streams sequentially into a single byte stream. The Secure Hash Algorithm 256 function takes an arbitrary length byte stream as input and outputs a fixed 256-bit hash value. This is used to number the data blocks, with values ​​ranging from 1 to the total number of data blocks. For record numbering, the value range is 1 to... .

[0085] The recursive formula for calculating the parent node's hash value is as follows:

[0086] ;

[0087] In the formula, For the first The hash value of each parent node is a 256-bit binary sequence; For the first The hash value of the left child node of each parent node is a 256-bit binary sequence. For the first The hash value of the right child node of each parent node is a 256-bit binary sequence. The parent node number is 1, and its value ranges from the total number of parent nodes.

[0088] When determining data consistency using the root hash value, the location of differing data blocks is achieved through layer-by-layer comparison. The specific implementation method is the same as described above and will not be repeated here.

[0089] The specific implementation of step S5 is that each database node executes the improved Gossip protocol propagation task at a set time interval. The formula for updating the local data version vector is expressed as follows:

[0090] ;

[0091] In the formula, For the local data version vector, the first The version number of each node, dimensionless; The first one in the remote data version vector received from neighboring nodes The version number of each node, dimensionless; This is the node index, with values ​​ranging from 1 to the total number of nodes. , The total number of nodes in a multi-database synchronization topology; This is the function for finding the maximum value.

[0092] The criteria for identifying missing change records are described below:

[0093] ;

[0094] In the formula, This is the version number difference, dimensionless. When this value is greater than 0, it indicates that the local node is missing the version number. For partial change records of a node, it is necessary to request sequence numbers from neighboring nodes within the range of 1. The changed data, this range represents version numbers greater than and less than or equal to All change records.

[0095] The specific implementation of step S6 is that the target database node performs conflict detection when receiving synchronization data. The conflict resolution rule is based on a comprehensive judgment formula of global timestamp and business priority, as follows:

[0096] ;

[0097] In the formula, The final data values ​​to be retained; The first conflicting data value is extracted from the received synchronization message; For the second conflicting data value, read from an existing record in the local database; For data values The corresponding global timestamp, in milliseconds, is generated using a hybrid logic clock; For data values The corresponding global timestamp, in milliseconds; For data values The source node is a globally unique identifier, which is a 128-bit binary identifier. For data values The source node's globally unique identifier is a 128-bit binary identifier; comparison operations... and For timestamps, comparison is based on numerical value; for identifiers, comparison is based on lexicographical order. Lexicographical comparison involves comparing the binary identifier byte by byte until a different byte is found.

[0098] In circular dependency detection, the formula for calculating the cost of transaction rollback is as follows:

[0099] ;

[0100] In the formula, For matters The cost of rollback is dimensionless; For matters The execution time, in seconds, is calculated as the difference between the transaction start time and the current time. The calculation method is as follows: ,in This is the current system time, in seconds. For matters The start time, in seconds, is obtained from the start time recorded in the transaction log; This represents the maximum execution time of all transactions in the loop, in seconds, obtained by taking the maximum value of the execution times of all transactions in the loop. For matters The amount of modified data, in kilobytes, is obtained through transaction log statistics, calculated by accumulating transactions. The number of data records involved in all write operations; This represents the maximum amount of modified data across all transactions in the loop, expressed in kilobytes. It is obtained by taking the maximum value of the modified data across all transactions in the loop. and These are weighting coefficients, dimensionless, with default values ​​of 0.6 and 0.4 respectively.

[0101] In multi-level adaptive compression processing, the formula for dynamically adjusting the compression level is expressed as follows:

[0102] ;

[0103] In the formula, The current compression level is dimensionless and ranges from 1 to 9. The system CPU utilization is expressed as a percentage and is obtained in real time through the operating system monitoring interface. This is the lowest compression level, dimensionless, and defaults to 1. The maximum compression level is dimensionless and defaults to 9; 80% is the upper limit threshold for CPU utilization; and 40% is the lower limit threshold for CPU utilization.

[0104] The formula for estimating the compression ratio of dictionary encoding is expressed as follows:

[0105] ;

[0106] In the formula, The dictionary-encoded compression ratio is dimensionless. The size of the dictionary is dimensionless and has a value of 4096. The total number of words in the text field is dimensionless and is obtained by scanning the data block to be compressed. The average character length of the original text, in bytes, is obtained through statistical analysis of scanned data samples. The calculation method is as follows: ,in For the first The character length of each sample, in bytes. The total number of samples, Number the sample; is a base-2 logarithmic function used to calculate the number of bits required for the index; 8 is the number of bits per byte.

[0107] The formula for calculating the number of bytes in variable-length integer encoding is as follows:

[0108] ;

[0109] In the formula, The number of bytes required for encoding variable-length integers; dimensionless. The integer value to be encoded is dimensionless. It is a rounding function; This is a logarithmic function with base 128; 128 is the range of values ​​that can be represented by each byte, since each byte uses 7 bits to store the actual value. .

[0110] The specific implementation of step S7 is that after the target database node receives the compressed data, it performs the corresponding decompression operation. After decompression, the data is written according to the optimized transaction scheduling plan. The update formula for the local data version vector is expressed as follows:

[0111] ;

[0112] In the formula, For the source node in the local data version vector The corresponding version number is dimensionless. The version number carried by the current write transaction, dimensionless, extracted from the transaction message header; This is the source node identifier, with values ​​ranging from 1 to the total number of nodes. .

[0113] In periodic full reconciliation tasks, data integrity verification is achieved through root hash value comparison. When a difference is detected, a repair synchronization process is triggered. The formula for calculating the repair priority of the difference data block is as follows:

[0114] ;

[0115] In the formula, For the first Repair priority for each differential data block, dimensionless; For the first The business priority weight value of the table to which each data block belongs is dimensionless and ranges from 1 to 10. It is obtained through the configuration file. This is the current timestamp, in seconds. For the first The last synchronization timestamp of each data block, in seconds, can be retrieved from the synchronization log; The full reconciliation period is in seconds, and the value is 86400, which is 24 hours. and These are weighting coefficients, dimensionless, with default values ​​of 0.7 and 0.3 respectively; 10 is the maximum value for the business priority weight.

[0116] The formula for generating the global pattern version number assigned by the pattern management system is as follows:

[0117] ;

[0118] In the formula, This is the global mode version number, represented by a 64-bit integer, and is dimensionless. The timestamp in seconds of the mode change occurrence, obtained via the system clock; It is an incrementing sequence number within the same second, dimensionless, with an initial value of 0, and incremented by 1 each time a mode change occurs within the same second; This is the shift factor, with a value of 4294967296, used to shift the timestamp to the high 32 bits.

[0119] The global timestamp is generated using a hybrid logic clock, and its calculation formula is as follows:

[0120] ;

[0121] In the formula, This is a mixed logic clock value, dimensionless. This is the current physical clock timestamp, in milliseconds, obtained from the system clock. The maximum logical timestamp recorded locally, in milliseconds, with an initial value of 0, and updated each time a local timestamp is generated or a remote timestamp is received. This is a logic counter, dimensionless, that increments when the physical clock is the same, with an initial value of 0 and a value range of 0 to 1. That is, 65535; This is a function to find the maximum value. This is the shift factor, with a value of 65536, used to embed the logic counter into the lower 16 bits.

[0122] When a mixed logic clock receives a remote event The formula for updating the local hybrid logic clock is as follows:

[0123] ;

[0124] In the formula, The updated local mixed logic clock value is dimensionless. The received remote mixed logic clock value is dimensionless and extracted from the remote event message header; This is a floor function used to extract the physical time portion of a remote clock. This indicates that the logic counter is incremented, when the physical time portion of the calculation result is... Logical timestamp of the last local record If the values ​​are the same, the logic counter is incremented by 1; otherwise, the logic counter is reset to 0 and then incremented by 1.

[0125] To better understand and implement this invention, a specific application scenario is provided below as Example 2: A technical team is responsible for building a cross-regional e-commerce transaction system. This system deploys independent MySQL database instances in Beijing, Shanghai, and Shenzhen, each processing order data for its respective region. To ensure users can query complete order information in any region, real-time bidirectional synchronization between the three database nodes is required. The team uses the multi-database automatic synchronization method proposed in this invention to build the synchronization system, solving the problem that traditional master-slave replication schemes cannot support multi-master writes and schema evolution.

[0126] During the initialization phase, the technical team assigned globally unique identifiers to the three database nodes: 0x010100000001 for the Beijing node, 0x010200000002 for the Shanghai node, and 0x010300000003 for the Shenzhen node. The first 8 bits represent the geographic location code, the middle 4 bits represent the MySQL type code, and the remaining bits are an incrementing sequence number. A connection graph was established between the nodes, connecting the Beijing node to both the Shanghai and Shenzhen nodes, and vice versa, forming a fully connected topology. The schema version number of all three nodes was initialized to 1, and a data version vector of length 3 was created with an initial value of [0, 0, 0]. A change data capture module was deployed on each node. This module monitors the order table for add, delete, and modify operations in real time by parsing the MySQL binlog files.

[0127] like Figure 2 As shown, the change data capture module detected an insert operation in the Beijing node's order table. The order number is Order20250501001, the order amount is 1580 yuan, and the order time is 10:32:15 AM on May 1, 2025. The module extracts the change record of this insert operation, including the operation type INSERT, the target table name orders, the primary key value Order20250501001, the data before the change being empty, and the data after the change containing all field values ​​of the order. The extracted data is encapsulated into a unified format synchronization message. The message header is filled with the source node identifier 0x010100000001, and a global timestamp of 1746067935256 is generated using a hybrid logical clock, where the physical clock part is 1746067935 seconds, the logical counter part is 256, the message sequence number is 10523, and the message type is incremental synchronization. The message body contains the complete field values ​​of the order, and character type fields are converted according to UTF-8 encoding. The generated, uniformly formatted synchronization messages are pushed to the Kafka message queue to await scheduling and processing.

[0128] The intelligent scheduling optimization model reads a set of transactions to be synchronized from the Kafka queue. The current batch contains a total of 128 transactions from three nodes. The model input layer receives the read / write set information for each transaction. For example, transaction T1 reads the record with the primary key Order20250501001 in the order table and updates the payment status field, while transaction T2 reads the delivery address field of the same order. The encoding layer calculates the correlation between transactions using a multi-head attention mechanism, identifying a read-after-write dependency between transactions T1 and T2. The feature extraction layer uses a three-layer graph convolutional neural network to process the dependency graph. Each graph convolutional layer aggregates the features of neighboring nodes and applies the ReLU activation function. The dynamic programming optimization layer executes a transaction reordering algorithm, analyzing the read / write sets of the 128 transactions to construct a dependency graph containing 128 nodes and 87 dependency edges. The dynamic programming state is defined as the combination of the scheduled transaction set and the current time point, with a state space size of... The actual search space is reduced to [a smaller size] through pruning strategies. The process involves enumerating candidate transactions whose predecessor dependencies have been satisfied during state transition, calculating incremental costs, and taking into account the estimated transaction execution time of 35 milliseconds, the probability of lock conflicts with already scheduled transactions (0.18 based on historical statistics), CPU resource utilization of 12%, and memory usage of 256MB. The transaction with the lowest cost is selected and added to the scheduling sequence. 23 independent transactions are identified and grouped into parallel batches, which can be executed simultaneously. The state transition is recursively executed to generate the final scheduling plan, dividing the 128 transactions into 16 serial stages and 23 parallel batches. Based on the difference between Beijing node mode version 1 and Shanghai node mode version 2, the output layer queries the mode management system to obtain mode change records between versions 1 and 2, discovering a new `customer_level` field in the order table, which is an integer with a default value of 0. Data conversion rules are generated, and the default value filling logic for the `customer_level` field is inserted into the conversion rule mapping table, as shown in Table 1.

[0129] Table 1 Data Transformation Rule Mapping Table

[0130] The entire table is divided into 32 data blocks based on the primary key range of the order table. Each data block contains approximately 5000 records within a consecutive primary key range. For example... Figure 3As shown, records within each data block are sorted in ascending order by primary key and serialized into a byte stream. The SHA-256 hash value of the byte stream is calculated as the hash of the leaf node. The hash values ​​of two adjacent leaf nodes are concatenated, and the SHA-256 hash is calculated again to obtain the parent node hash. This process is repeated to construct a 5-level Merkle tree structure. The root hash values ​​of the order tables in the Beijing and Shanghai nodes are 0x8f3b2a1c and 0x8f3b2a1c respectively. Comparison shows they are the same, indicating complete data consistency and no synchronization is needed. The root hash values ​​of the order tables in the Beijing and Shenzhen nodes are 0x8f3b2a1c and 0x7e4c3b2d respectively. Comparison shows they are different. Further comparison of the hash values ​​of the left and right child nodes of the root node leads to the third branch of the right subtree, where the hash value differs. Recursively traversing this branch, the primary key range of the data block corresponding to the leaf node is finally located to Order20250501001 to Order20250501156. This data block has a difference and requires synchronization.

[0131] The Gossip protocol propagation module has been improved to periodically perform propagation tasks on each node. The Beijing node maintains a list of neighboring nodes containing the network addresses of the Shanghai and Shenzhen nodes, with a local data version vector of [523, 478, 501], indicating that the Beijing node knows it has performed 523 operations, knows the Shanghai node has performed 478 operations, and knows the Shenzhen node has performed 501 operations. The Beijing node randomly selects three neighboring nodes at 200-millisecond intervals; in this round, it selects the Shanghai and Shenzhen nodes. A propagation message is sent to the Shanghai node, containing the local data version vector [523, 478, 501] and a summary of the last 50 change log entries. The summary contains sequence numbers from 10523 to 10573, operation types including 12 INSERT, 23 UPDATE, and 15 DELETE, and timestamps ranging from 1746067935 to 1746067955 seconds. The system receives the remote data version vector [518, 512, 496] returned by the Shanghai node. Element-by-element comparison reveals that the first position of the local vector, 523, is greater than the remote vector 518, indicating that the Shanghai node is missing some changes from the Beijing node. Six change records with sequence numbers 518 to 523 are pushed to the Shanghai node, including three order addition operations and three payment status update operations. Upon receiving the change records, the Shanghai node merges them into its local database, executes three INSERT statements and three UPDATE statements, and updates the first position of the local data version vector to 523.

[0132] When the target database node receives synchronized data, it performs conflict detection. The Shenzhen node receives an update message for Order20250501001 from the Beijing node. This message changes the payment status from pending payment to paid, with a global timestamp of 1746067935256. The Shenzhen node already has a record for this order locally, with a payment status of pending payment and a global timestamp of 1746067935230. Comparing the two timestamps, the received message's timestamp is larger, indicating the received data is a newer version, and the received data is used to update the local record. While processing 128 transactions from the three nodes, the Shenzhen node detects a circular dependency: transaction T15 is waiting for a row lock on the order table held by transaction T28; transaction T28 is waiting for a row lock on the user table held by transaction T42; and transaction T42 is waiting for a row lock on the inventory table held by transaction T15. The Shenzhen node reports the lock wait information to the distributed deadlock detection service, which constructs a global wait directed graph containing 128 transaction nodes and 256 waiting edges. A topological sort was performed on the directed graph, revealing a loop formed by nodes T15, T28, and T42. The costs of the three transactions were calculated: transaction T15, having executed for 120 milliseconds and modified 8 records, had a cost of 128; transaction T28, having executed for 85 milliseconds and modified 5 records, had a cost of 90; and transaction T42, having executed for 160 milliseconds and modified 12 records, had a cost of 172. The transaction with the lowest cost, T28, was selected for rollback. A rollback instruction was sent to the Shanghai node holding the transaction. The Shanghai node rolled back transaction T28 and released the user table row lock, breaking the circular dependency.

[0133] like Figure 4As shown, multi-level adaptive compression processing is performed on the synchronization data to be transmitted. Analysis of this batch of synchronization data reveals 256KB of data, of which text fields such as order remarks and delivery addresses account for 68%, numeric fields such as order amount and quantity account for 25%, and binary fields such as order voucher images account for 7%. Dictionary encoding is used for text fields. A dictionary table containing 4096 high-frequency words is built by scanning all text fields. These high-frequency words include common terms such as receiving, delivery, and payment. The original text is replaced with index numbers; the original text "Order remarks, please deliver as soon as possible" is encoded as the index sequence [1523, 2047, 1892]. Run-length encoding is performed on the index sequence, revealing that index 1523 appears 5 times consecutively, encoded as a combination of the count value 5 and the index value 1523. Differential encoding is used for numeric fields. The adjacent differences in the order amount sequence [1580, 1620, 1595, 1610, 1605] are calculated to be [40, -25, 15, -5]. Variable-length integer encoding is used for the difference values. A difference of 40 is represented in binary as 00101000. Values ​​less than 128 can be represented using 1 byte, resulting in an encoding of 0x28. The absolute value of the difference -25, 25, is represented in binary as 00011001, encoded as 0x19 with a sign bit. Binary fields are transmitted directly without compression. The monitoring system's current CPU utilization is 45%, below the 40% threshold. Therefore, the compression level is increased to level 8, using more computation to achieve a higher compression ratio. After compression, the data size is reduced to 89KB, achieving a compression ratio of 65%. The compressed data is transmitted to the target node via TCP, reducing network bandwidth usage from the original 256KB to 89KB.

[0134] After receiving the compressed data, the Shenzhen node performs decompression. Based on the compression type dictionary encoding and run-length encoding indicated in the message header, it first reads the dictionary table, restoring the index sequence [1523, 2047, 1892] to the original text "Order Remarks: Please Deliver As Soon as Possible." The run-length encoded data is expanded, restoring the count value 5 and index value 1523 to five consecutive indices of 1523. For variable-length integer encoded data, it reads byte by byte, with the highest bit of the first byte (0x28) indicating the end and the lower 7 bits representing the value 40. The difference sequence [40, -25, 15, -5] is read and accumulated to restore the original numerical sequence [1580, 1620, 1595, 1610, 1605]. After decompression, data is written according to the optimized transaction scheduling plan, which specifies 16 serial stages and 23 parallel batches. For the first parallel batch containing 23 transactions, 23 concurrent threads are started to simultaneously write to different primary key ranges in the order table, with each thread responsible for writing approximately 5 records. Transactions with dependencies are executed strictly in serial order; transaction T1 must complete before transaction T2 to satisfy the read-after-write dependency constraint. The local data version vector is updated after each transaction's write operation is completed. After completing six transactions from the Beijing node, the first position of the local data version vector is updated from 518 to 524.

[0135] The system initiates a full reconciliation task every 24 hours. This task uses a Merkle tree hierarchy to compare the integrity of order table data across the three nodes. The root hash value for the order table in the Beijing node is calculated to be 0x9a5e4f3b, the Shanghai node's root hash value is 0x9a5e4f3b, and the Shenzhen node's root hash value is 0x8b4d3c2a. A difference was found in the Shenzhen node's root hash value. Further comparison of child node hash values ​​was used to locate the discrepancy branch, ultimately pinpointing a difference in the data block within the primary key range Order20250503001 to Order20250503156. This triggered a repair synchronization process, re-executing change data capture to extract the latest data for this data block, totaling 156 order records. These 156 records were encapsulated into a unified format synchronization message, and after conflict detection, written to the Shenzhen node's database. After writing, the Merkle tree root hash value for the Shenzhen node was recalculated to 0x9a5e4f3b, matching that of the Beijing and Shanghai nodes, thus completing the data repair.

[0136] like Figure 5As shown, performance monitoring data for the entire synchronization process indicates an average system synchronization latency of 156 milliseconds. After transaction scheduling optimization, the parallel execution ratio reached 18%. Data compression reduced network transmission volume by 65%. Change propagation via the Gossip protocol covered all nodes within 500 milliseconds, and data inconsistencies detected by periodic reconciliation tasks were automatically repaired within 15 minutes. The technical team successfully solved the real-time consistency problem of cross-regional order data using the multi-database synchronization system built using this invention, meeting the business needs of e-commerce platforms. It should be noted that detailed explanations of the variables involved in this invention are shown in Tables 2 and 3.

[0137] Table 2. Variable Explanation Table (Part 1)

[0138] Table 3. Variable Explanation Table (Part Two)

[0139] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatic synchronization of multiple databases under an AI platform, characterized in that, Includes the following steps: Initialize the multi-database synchronization topology, assign a globally unique identifier to each database node, establish a connection graph between nodes, record the schema version number and data version vector of each database node, and start the change data capture module. The change data capture module monitors the add, delete, and modify operations of each database node in real time, and converts the captured change records into a unified format synchronization message. The unified format synchronization message is input into the intelligent scheduling optimization model, which performs dependency analysis on the transactions to be synchronized and executes a transaction reordering optimization algorithm based on dynamic programming to generate an optimized transaction scheduling plan. Based on the difference between the schema version number corresponding to the source node identifier and the schema version number corresponding to the target node in the optimized transaction scheduling plan, data transformation rules are dynamically generated. The synchronized data after data transformation is divided into multiple data blocks according to the primary key range, and a hash value is calculated for each data block to construct a Merkle tree hierarchical structure. Start the improved Gossip protocol propagation module; The target database node performs conflict detection when receiving synchronization data. The target database node decompresses the received compressed synchronous data, executes an optimized transaction scheduling plan to complete the data writing and update the local data version vector, and performs a periodic full reconciliation task.

2. The method according to claim 1, characterized in that, The globally unique identifier is a 128-bit binary identifier consisting of the node deployment geographic location code, the node type code, and an incrementing sequence number.

3. The method according to claim 2, characterized in that, The data version vector is an ordered array that records the latest operation version number of each database node. The length of the ordered array is equal to the total number of nodes in the multi-database synchronization topology.

4. The method according to claim 3, characterized in that, The captured change records are converted into a unified format synchronization message. Specifically, the operation type, target table name, primary key value, data before the change, and data after the change are extracted from the change records and encapsulated into a unified format synchronization message structure containing a message header and a message body.

5. The method according to claim 4, characterized in that, The structure of the intelligent scheduling optimization model is as follows: the input layer receives the read and write set information and timestamp information of the set of transactions to be synchronized; the encoding layer extracts the dependency relationship features between transactions through a multi-head attention mechanism; the feature extraction layer uses a graph convolutional neural network to learn node features of the directed graph constructed by the dependency relationship; the dynamic programming optimization layer executes a transaction reordering optimization algorithm based on dynamic programming based on the learned feature vectors; and the output layer generates an optimized transaction scheduling plan that includes the transaction execution order and parallel batch partitioning.

6. The method according to claim 5, characterized in that, The transaction reordering optimization algorithm based on dynamic programming specifically analyzes the read and write sets of each transaction in the set of transactions to be synchronized to identify data dependencies, constructs a directed acyclic dependency graph, defines the dynamic programming state as the combination of the set of scheduled transactions and the current time point, and selects all transactions whose predecessor dependencies have been satisfied as candidate transactions from the unscheduled transactions during state transition, and recursively executes state transitions until all transactions are scheduled.

7. The method according to claim 6, characterized in that, Data conversion rules are dynamically generated based on the difference between the mode version number corresponding to the source node identifier and the mode version number corresponding to the target node in the optimized transaction scheduling plan. Specifically, the current mode version numbers of the source database and the target database are queried, all mode change records between the two mode version numbers are retrieved from the mode management system, and data conversion rules are generated based on the mode change records.

8. The method according to claim 7, characterized in that, Constructing a Merkle tree hierarchy involves dividing the entire table into several data blocks based on the primary key range. For each data block, all records are sorted according to the primary key range and concatenated into a byte stream. The SHA-256 hash value is then calculated to serve as the hash of the leaf node. The hash values ​​of two adjacent leaf nodes are concatenated and the SHA-256 hash value is calculated again to serve as the hash of the parent node. This concatenation and hash calculation process is repeated until a unique root hash value is generated.

9. The method according to claim 8, characterized in that, Data consistency is determined by comparing the root hash values ​​of the source and target databases and locating the differing data blocks. If the root hash values ​​are different, the hash values ​​of the left and right child nodes of the root node are compared. The process is recursively traversed downwards through the subtree branches with different hash values ​​until the leaf node hash is reached. The primary key range of the data blocks corresponding to the hash values ​​of all leaf nodes with different hash values ​​is recorded.

10. The method according to claim 9, characterized in that, The steps of the improved Gossip protocol propagation module are as follows: each database node maintains a list of neighbor nodes and a local data version vector. The database node randomly selects a fixed number of neighbor nodes from the list of neighbor nodes at set time intervals and sends a message containing the local data version vector and a summary of the latest change log to the selected neighbor nodes.

Images