A data synchronization method and system with full-link backpressure characteristics

Abstract

The application discloses a data synchronization method and system with full-link back pressure characteristics, comprising: configuring metadata of a synchronization task; starting incremental synchronization of a binlog log, and selectively performing full synchronization; routing generated add, delete, modify and query events to a distributed parallel Stream pipeline; and distributing the events to an Actor entity in an Actor model shard cluster according to dimension fields configured for each table in an asynchronous request response mode; performing event persistence and state updating, and complex conversion logic based on the current state, and returning the result to the Stream pipeline; performing message batching, sorting and deduplication processing on Response events by the Stream pipeline; writing the result to a Sink data source, and performing an asynchronous ACK mechanism to ensure reliable synchronization. The application can quickly realize complex model synchronization conversion between heterogeneous data sources through highly configurable synchronization tasks, and has the characteristics of short link, few components and efficient synchronization.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and specifically to a data synchronization method and system with full-link backpressure characteristics. Background Technology

[0002] In the internet information age, business scenarios and needs are complex and ever-changing, and a single database can no longer meet business requirements. Databases have presented a situation where both SQL and NoSQL are flourishing. However, this has brought about the problem of data flow and synchronization between different storage systems, due to the following complexities in data synchronization:

[0003] Synchronizing heterogeneous data sources is more complex than synchronizing homogeneous data sources. Synchronizing homogeneous data sources is relatively simple due to their similar design philosophies and models. However, the optimal data models for heterogeneous data sources differ significantly. For example, relational databases like MySQL, following normalization, generate a large number of business tables; while search engines like Elasticsearch or Apache Solr excel at full-text search of large, wide tables or nested JSON, but are not adept at multi-table joins on massive datasets.

[0004] Ensuring data synchronization consistency is crucial. Guaranteeing absolute error-free operation is difficult, and the need to repair data is inevitable. A complex challenge lies in synchronizing existing data and fine-grained repair of business data in a non-blocking manner (without affecting incremental synchronization).

[0005] In data synchronization systems, the synchronization pipeline of synchronization tasks presents a significant single point of failure, making the implementation of primary / backup synchronization pipelines and automatic failover highly complex. How can the stable operation of the synchronization system be guaranteed under the premise of high throughput and low latency? Summary of the Invention

[0006] This invention provides a data synchronization method and system with end-to-end backpressure characteristics to solve the problems of large differences in optimal models of heterogeneous data sources, difficulty in ensuring data consistency, poor data synchronization stability and low efficiency in the data synchronization process. It achieves near real-time automatic data synchronization and ensures data integrity and accuracy.

[0007] A data synchronization method with end-to-end backpressure characteristics, characterized by comprising the following steps:

[0008] S1. Configure the metadata for the synchronization task. The metadata includes: connection configuration of the Source data source and Sink data source, database tables for log parsing, and calculation and transformation logic of the Actor model sharded cluster.

[0009] S2. Use the metadata from step S1 to start a synchronization task, perform incremental synchronization based on the database binlog log, generate add, delete, and modify events after synchronization, perform full synchronization based on SQL mode, and generate query events after synchronization.

[0010] S3. Route the CRUD events generated in step S2 to a distributed parallel Stream pipeline;

[0011] S4. Parallel Stream pipelines distribute CRUD events to Actor entities in the Actor model sharded cluster in an asynchronous request-response mode, based on the dimension fields configured in each table.

[0012] S5. In the Actor entities of the Actor model sharded cluster, persist events and update states according to the order of CRUD events, as well as perform complex transformation logic based on the current state, and return the results to the Stream pipeline;

[0013] The S6.Stream pipeline performs message batching and sorting / duplicate processing on Response events.

[0014] S7. Write the result of step S6 to the Sink data source (such as Elasticsearch, RabbitMQ, Kafka, etc.) and perform an end-to-end asynchronous ACK mechanism between the Source data source and the Sink data source. If the ACK is successful, the reliable data synchronization is completed.

[0015] Step S2, the full synchronization based on SQL mode, specifically includes:

[0016] The full synchronization task submitted via S2.1.API includes multi-table join retrieval tasks and multi-table independent retrieval tasks;

[0017] S2.2. If the task in step S2.1 involves multi-table joins, the submitted interface data will have one main table and multiple auxiliary tables. SQL is constructed for the main table, and sharding logic is applied to obtain sharded SQL. The sharded SQL of the main table is pulled in parallel and under controlled conditions. The main table records drive the reverse lookup of the auxiliary tables. The data obtained from the query is formatted to maintain consistency with the data format of incremental synchronization. The data in the main table and auxiliary tables are processed by message batching (the purpose is to minimize the frequent switching of the sharded Actor entity state between disk and memory in step S5, so that N messages only need to be calculated once in the Actor entity in the Actor model sharding cluster) to obtain the final data.

[0018] S2.3. If step S2.1 involves a multi-table independent fetching task, then construct the table SQL, distribute it evenly to each node, perform sharding logic on the table SQL to obtain sharded SQL, perform parallel controlled fetching on the sharded SQL, convert the format of the fetched data to maintain consistency with the incrementally synchronized data format, and obtain the final data.

[0019] S2.4. Use the final data from step S2.2 or the final data obtained from S2.3 as the query event.

[0020] In step S2, the CRUD events generated by the incremental synchronization of the database binlog all contain uid and ts attributes. ts represents the log generation time (millisecond timestamp) in the database binlog, while uid is a Long type value composed of an incrementing number in the binlog filename suffix (plus a configurable compensation parameter, defaulting to 0 to handle database log reset scenarios) and the byte offset in the binlog. This value is incrementing and uniquely identifies the CRUD operation on any row in the database table. When a full synchronization task is triggered, the latest uid and ts recorded periodically by incremental synchronization will be used as the uid and ts values ​​for all query events in that batch of full synchronization. Based on uid and ts, the order of CRUD events can be accurately determined.

[0021] In step S5, the complex transition logic based on the current state includes:

[0022] S5.1. The current state of the Actor entity in the Actor model sharded cluster consists of multiple sub-tables obtained by logical partitioning the dimension fields in step S4;

[0023] S5.2. Operations on one or two sub-tables in the current state are encapsulated into flexible SQL. After SQL calculation and processing, a temporary intermediate table is obtained, which can be used as the input table for the next operation.

[0024] S5.3. The table-level pipeline mechanism can combine multiple operations to obtain arbitrarily complex transformation logic. A pipeline consists of at least one operation, and operations without dependencies can be executed concurrently to accelerate computation.

[0025] In step S7, the end-to-end asynchronous ACK mechanism from the Source data source to the Sink data source can strictly guarantee at least one message delivery semantics. Specifically, it includes:

[0026] In step S2, the CRUD (Create, Read, Update, Delete) events generated by incremental or full synchronization all have an ActorRef address and a unique AckId attribute for message confirmation. In step S7, after data is successfully written to the Sink data source, end-to-end asynchronous ACK processing from the Source data source to the Sink data source is performed based on the ActorRef and AckId. If confirmation fails (ACK message lost or timed out), incremental synchronization will quickly restart and reset to the most recently successfully ACKed point to continue synchronization; while full synchronization will continue synchronization, retaining unconfirmed messages for retry after full synchronization is completed.

[0027] This invention also provides a data synchronization system with end-to-end backpressure characteristics, characterized in that it includes:

[0028] The task metadata module is used to configure and persist task metadata;

[0029] A full data synchronization module for efficient parallel synchronization of existing data;

[0030] An incremental data synchronization module that extracts and parses binlog logs from mainstream relational databases (MySQL, PostgreSQL, TiDB, PolarDB, Oracle, etc.);

[0031] A data sink module that supports the persistence of final results (supporting Elasticsearch, Kafka, RabbitMQ, MySQL, etc.);

[0032] A distributed management module for massive Actor entities, with configurable Actor computation logic and a sharded Actor cluster management module.

[0033] A dynamic parallelism management module for managing synchronous parallelism;

[0034] A timed data inspection module used to ensure data integrity and accuracy.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] First, because the CRUD events in step S2 all include uid and ts attributes used to determine the order of events, the flow and processing of downstream events do not need to be strictly ordered, but the eventual consistency of the synchronized data can still be guaranteed. In this way, the synchronization link can not only perform full synchronization in a non-blocking binlog incremental synchronization manner, but also fully perform asynchronous concurrent processing of events, as well as dynamically adjust the parallelism of the parallel stream pipeline.

[0037] Second, thanks to the Actor model sharding cluster used in steps S4 and S5, the lock-free concurrency characteristics of the Actor model make it easy to perform computational processing of massive amounts of data. The passivation mechanism of the Actor can be used to achieve separation of hot and cold data and efficient use of memory.

[0038] Third, the complex transformation logic based on the current state in step S5 is configurable (persistent in the task metadata). Through the table-level pipeline mechanism and customizable SQL operations, it can complete the complex aggregation transformation between multiple tables, and realize the transformation of different data models between heterogeneous data sources (such as synchronizing a relational database to a search engine).

[0039] Fourth, the end-to-end asynchronous ACK mechanism from the Source data source to the Sink data source in step S7 further guarantees at least one message passing semantics and eventual data consistency.

[0040] Fifth, the synchronization method of the present invention is completely decoupled from the business code (only depends on the business database), and benefits from the implementation of a synchronization pipeline (such as AkkaStream) that follows the ReactiveStream standard. Steps S2 to S7 have full-link backpressure characteristics, and have the advantages of short synchronization links, fewer components involved, high resource utilization, and not being highly dependent on message middleware. Attached Figure Description

[0041] Figure 1 This is a flowchart of the synchronization method of the present invention;

[0042] Figure 2 This is a flowchart of the full synchronization based on SQL mode described in step S2 of the synchronization method of the present invention;

[0043] Figure 3 This is a schematic diagram of the running state of a single synchronous task implemented in this invention;

[0044] Figure 4 This is a diagram illustrating the running state of a single stateless synchronization task (without requiring aggregation and transformation across multiple tables). Detailed Implementation

[0045] During implementation, message middleware is not mandatory, and the synchronization link does not require message middleware to buffer data. This is mainly because the entire synchronization link from Source to Sink is implemented in full accordance with the ReactiveStream standard specification. The downstream sends demand signals to the upstream, and the upstream sends data to the downstream according to the demand signals. There is a dynamic coordination mechanism between upstream and downstream production and consumption, which can effectively utilize memory resources to achieve stable data synchronization and backpressure transmission.

[0046] like Figure 1 , Figure 2 As shown, a data synchronization method with end-to-end backpressure characteristics includes the following steps:

[0047] S1. Configure the metadata for the synchronization task. The metadata includes: connection configuration of the Source and Sink data sources, database tables for log parsing, and computation and transformation logic of the Actor model sharded cluster.

[0048] Task metadata can be persisted using the etcd component. The metadata of a synchronization task consists of a set of key-value pairs, where the value stores data in JSON or HOCON format. For example, the key-value format for the metadata of the S1 step synchronization task is as follows:

[0049]

[0050]

[0051] Similarly, the configuration for synchronized database tables corresponds to the key in etcd.<task_id> / table_source, the corresponding configuration on the data sink side is<task_id> / table_sink, the corresponding conversion logic configuration is stored in<task_id> / pipeline_sql, the metadata for the sharded Actor cluster state backend configuration is stored in<task_id> ` / complextable_event_source`, etc., here, the metadata of a synchronization task is split into a set of key-value pairs as much as possible to avoid the problem of excessively large data stored in the value (V) of a single key-value pair affecting read and write efficiency. As for the JSON or HOCON content in the value (V), it is entirely flexibly defined by the business requirements of the synchronization system.

[0052] S2. Use the metadata from step S1 to start the synchronization task, performing incremental synchronization based on database binlog logs and full synchronization based on SQL mode;

[0053] Incremental synchronization based on binlog logs uses a custom Source approach (supported by backpressure stream implementations conforming to the ReactiveStream standard, such as Akka Stream and Project Reactor). It can integrate mature open-source components (Canal, Debezium, Maxwell, TiCDC) to extend to Canal Source, Debezium Source, Maxwell Source, TiCDC Source, etc. These custom Source components need to reliably store the binlog log consumption offset. Optional components include Etcd, Zookeeper, or direct storage in the source database. Furthermore, the asynchronous ACK mechanism from the Source to the Sink in step S7 requires the cooperation of the custom Source. Only after all previous points in the custom Source have been asynchronously ACKed can the consumption offset of the log be committed; otherwise, it indicates possible message loss or ACK confirmation timeout, requiring a reset of the consumption points (or a restart). Figure 3 In the Actor model sharding cluster named Binlog Parser ShardActor, the corresponding Actor ensures that potentially lost messages are resent downstream, guaranteeing at-least-once message delivery semantics. These custom Sources are provided by the stateless Actor model sharding cluster (such as...). Figure 3 The BinlogParserShard Actor provides one-to-one hosting for Actor entities. By leveraging the characteristics of sharded Actor clusters, custom Sources can be evenly distributed across cluster nodes, enabling parallel parsing and subscription of database logs. Furthermore, custom Sources can achieve fault tolerance and recovery capabilities through the Actor restart mechanism.

[0054] Full data synchronization based on SQL mode can leverage the characteristics of SQL to adapt to most mainstream relational databases. The main process of full data synchronization is described below. Figure 2 Specifically, it includes:

[0055] S2.1. Based on the full synchronization task submitted by the API, it can be distinguished whether it is an independent retrieval of multiple tables or a retrieval of multiple tables with related data.

[0056] S2.2. If the task in step S2.1 involves multi-table joins, the submitted interface data will consist of one main table and multiple auxiliary tables. SQL is constructed for the main table, and sharding logic is applied to obtain sharded SQL. The sharded SQL for the main table is then pulled in parallel under controlled conditions. The main table records drive a reverse lookup of the auxiliary tables. The retrieved data is formatted to maintain consistency with the incrementally synchronized data format. Finally, the final data from the main and auxiliary tables undergoes N:1 message batching processing (the purpose is to minimize the frequent switching of the sharded Actor entity state between disk and memory in step S5, so that N messages only require one state calculation in the Actor).

[0057] S2.3. If step S2.1 involves a multi-table independent fetching task, then construct the table SQL, distribute it evenly to each node, perform sharding logic on the table SQL to obtain sharded SQL, perform parallel controlled fetching on the sharded SQL, and convert the format of the fetched data to maintain consistency with the data format of incremental synchronization.

[0058] Steps S2.2 and S2.3 both involve partitioning the table based on index columns to construct SQL for a partitioned table with a reasonable data volume. If the user has explicitly configured the partitioning column, the partitioning is performed accordingly. If the user has not configured the partitioning column, the system can automatically identify the appropriate index columns and their types in the table, process them using appropriate partitioning logic, and obtain the partitioned SQL. Subsequently, uncompressed substreams conforming to the ReactiveStream standard can be used to achieve parallel and controlled data retrieval from the partitioned SQL. To ensure that the partitioned SQL executes normally in the data source, the cursor query feature available in most mainstream relational databases needs to be fully utilized to avoid client memory overflow issues.

[0059] For synchronizing massive amounts of existing data, steps S2.2 and S2.3 are time-consuming and prone to partial failures and interruptions. To quickly resume synchronizing the remaining unfinished shards, a full synchronization with a shard-level checkpoint mechanism is required. The checkpoint mechanism maintains and periodically updates the synchronization status of each shard. This status mainly records the task submission and completion times, the corresponding UID and TS information, the total number of shards in the table, the total number of records in the table, the number of successfully synchronized records, the set of records with unacknowledged ACKs (each element being a uniquely identified ackId), and the list of shards (shard metadata mainly includes the shard SQL and whether the shard synchronization is complete).

[0060] S2.4. Combine the data obtained in step S2.2 or S2.3 with the data obtained from incremental synchronization into the parallel Stream pipeline.

[0061] Full data synchronization and incremental data synchronization share a parallel Stream pipeline and can occur simultaneously. Eventual consistency is not guaranteed by the absolute order of message events, but by the `uid` and `ts` fields of the message events themselves (these two fields strictly determine the message order). `ts` is a millisecond timestamp representing the creation time of the log in the database binlog. `uid` is a Long type value composed of an incrementing number in the binlog filename suffix (plus a configurable compensation parameter, defaulting to 0 to handle database log reset scenarios) and the offset in the binlog. This value uniquely identifies the CRUD operation on any row in the database table and is guaranteed to be incrementing. During incremental synchronization, the most recently asynchronous end-to-end ACK-confirmed `uid` and `ts` position information is periodically committed. When a full synchronization task is triggered, the latest incremental `uid` and `ts` are used as the `uid` and `ts` values ​​for all existing data synchronized in that batch.

[0062] The message order is determined by using uid and ts, allowing for full asynchronous parallel concurrency in the backpressure stream. Only during asynchronous Request / Response communication between the backpressure stream and the ComplexTableShard Actor cluster does the uid and ts fields determine whether the Actor state in the ComplexTableShard Actor cluster needs updating. The Actor state maintains a miniature database (data from multiple tables involved in a business entity), and also includes a version field. Leveraging the serial nature of the Actor model, the version field is incremented by 1 whenever the Actor state changes. The Response returned from the ComplexTableShard Actor cluster to the parallel Stream pipeline can then use this version value to determine the message order. Finally, the Sink uses this version value for optimistic locking, ensuring eventual consistency during concurrent writes.

[0063] S3. Route the CRUD (Create, Read, Update, Delete) events generated in step S2 to a distributed parallel Stream pipeline;

[0064] Based on the uid and ts attributes of events, it is not necessary to strictly guarantee the order of messages in the Stream pipeline. Therefore, event routing can adopt either a round-robin routing strategy or a routing strategy based on the fastest consumer.

[0065] Parallel Stream pipelines consist of stateless Actor model sharded clusters (such as...) Figure 3In DynamicParallismShardActor, the Actor entities are managed one-to-one, which can make full use of the important features of sharded Actors, such as automatic sharding and fault tolerance mechanisms, to achieve dynamic parallelism management of synchronous tasks.

[0066] S4. Parallel Stream pipelines distribute CRUD events asynchronously in Request / Response mode to the abstracted sharded Actor entities according to the dimension fields configured in each table;

[0067] In the S5.Actor model sharded cluster, event persistence and state updates are performed on the sharded Actor entities, as well as complex transformation logic based on historical states, and the results are returned to the Stream pipeline;

[0068] In step S4, the sharded Actor entities in the Actor model sharded cluster determine the order of events based on the uid and ts of the Request event and the uid' and ts' of the historical data maintained in the Actor state. If the events are out of order, the Request event is ignored and an empty response is returned; otherwise, the event is persisted, the Actor state is updated, and an asynchronous calculation is performed based on the updated current state to return a Response. The Response contains a version field (the version is maintained by the Actor model and is strictly guaranteed to be monotonically increasing), which can also be used downstream to determine the order of Response responses.

[0069] The sharded Actor entities in the sharded cluster of the Actor model in steps S4 and S5 refer to, for example, Figure 3 The ComplexTableShard Actor sharding cluster (which can be implemented based on open-source frameworks like Akka or Orleans) is used. State partitioning is, in principle, based on the granularity of business entities (e.g., in an electronic signature scenario, all tables related to each contract can be placed in one Actor). The aim is to fully leverage the lock-free concurrency of the Actor model for massive state computations. Using stateful Actors as state containers (requiring backend support for Actor state storage, such as Cassandra, EventStore, or MySQL), optimal data model transformation for heterogeneous data sources can be achieved. Managed state is preferentially stored in memory to accelerate computation. To efficiently utilize memory, appropriate passivation (persisting state to external components to release memory) strategies need to be configured according to the business scenario. These can be time-based passivation strategies, custom business-based passivation strategies (e.g., in an electronic signature scenario, active passivation can be implemented upon contract completion), or a combination of both.

[0070] In a ComplexTableShard Actor sharded cluster, the Actor state aggregation logic is strongly business-dependent. To improve the generalization capability of the synchronization system, this aggregation calculation logic needs to be abstracted into a configurable form. An Actor in a ComplexTableShard Actor sharded cluster contains multiple original table data from a synchronized business entity, which can be treated as a miniature database. Using the Apache Calicite open-source component, a pipeline mechanism and customizable SQL capabilities can be abstracted, enabling flexible aggregation and transformation of the state in the Actor. The pipeline mechanism consists of at least one cascaded Operation. Each Operation takes one or two tables as input and outputs a new table. The specific operations of each Operation are described using customizable SQL. Further combination of the pipeline allows for the configuration of complex transformation logic.

[0071] Of course, introducing such Figure 3 Using a ComplexTableShardActor sharded cluster as a state container incurs costs. For simple synchronization tasks involving only intra-table data transformations (such as field type conversions, field renaming, etc., without complex cross-table aggregation transformations), there is absolutely no need to introduce a complex ComplexTableShardActor sharded cluster, such as... Figure 4 As shown, the advantage of this is that the synchronization task is lightweight and efficient, and the bottleneck of synchronization mainly depends on the performance of the data sink component and the parallelism of the stream pipeline.

[0072] The S6.Stream pipeline performs message batching and sorting / duplicate processing on Response events.

[0073] The fundamental purpose of batch messaging and sorting / deduplication is to optimize write performance and improve throughput. Especially in high-concurrency business scenarios, the same row of records in a table in the business database may be frequently modified in a short period of time, generating a large number of Row-mode binlog logs. This will result in a large number of redundant Response events downstream. As long as the latest Response event with the same unique identifier is written (other old Response events can be completely ignored), eventual data consistency can still be guaranteed.

[0074] S7. Write the results of step S6 to the Sink data source (such as Elasticsearch, RabbitMQ, Kafka, etc.) and perform an asynchronous ACK mechanism from the Source to the Sink.

[0075] The asynchronous ACK mechanism from the Source to the Sink ensures at-least-once message delivery semantics. During the flow of events from the Source to the Sink, each event has a message acknowledgment address (ActorRef) and a unique AckId attribute. Once data is successfully written to the Sink data source, asynchronous ACK processing from the Source to the Sink is performed based on the ActorRef and AckId. If acknowledgment fails (ACK message lost or timed out), incremental synchronization will quickly restart and reset the synchronization point to continue resuming synchronization, while full synchronization will continue synchronizing, retaining unacknowledged messages for retry after full synchronization is complete.

[0076] For physical or logical deletion events generated by upstream business processes, the Sink side handles them as logical deletions (soft deletions). The advantage of this is that, through optimistic locking, the version field is used to write or ignore messages, eliminating the need for absolute message writing order guarantees and allowing for full concurrent writing. This ensures eventual data consistency while also allowing for querying previous deletion history in the data source being written to.

[0077] To further ensure data integrity and accuracy, the synchronization method and system also include independent, scheduled data inspection and comparison tasks, which can proactively detect inconsistent data during periods of low system load. The scheduled data inspection task fully reuses the full data synchronization implementation by pulling data from the data source. By querying the metadata of the synchronization task, it performs data comparison and verification based on the status of each stage of the synchronization link (mainly three stages: Source, ComplexTableShard Actor, and Sink). When inconsistencies are found, the details of the inconsistency are recorded for easy subsequent querying and investigation of the cause. When the amount of erroneous data is small, automatic repair is possible; when the amount of data is large, the corresponding status can be recorded for manual intervention.

Claims

1. A data synchronization method with full link backpressure characteristics, characterized in that, The method comprises the following steps: S1. configuring metadata of a synchronization task, the metadata comprising: connection configuration of a Source data source and a Sink data source, library table of log analysis, and computing conversion logic of an Actor model sharded cluster; S2. starting the synchronization task by using the metadata of step S1, performing incremental synchronization based on a database binlog log, generating add-delete-modify events after synchronization, performing full-amount synchronization based on a SQL mode, generating query events after synchronization, and the add-delete-modify events and the query events forming add-delete-modify-query events; In step S2, the full-amount synchronization based on the SQL mode is performed, and the query events are generated after synchronization, and specifically comprises: S2.

1. The full-amount synchronization task submitted by the API comprises a multi-table associated pulling task and a multi-table independent pulling task; S2.

2. If the multi-table associated pulling task is submitted in step S2.1, the interface data submitted has one main table and multiple auxiliary tables, a SQL is constructed for the main table, sharding logic is performed to obtain sharded SQL, the sharded SQL of the main table is controlled and pulled in parallel, the auxiliary tables are inversely inquired by the main table record, the data obtained by the inquiry is format-converted to be consistent with the data format of the incremental synchronization, and the data of the main table and the auxiliary tables is processed in a message batch mode to obtain final data; S2.

3. If the multi-table independent pulling task is submitted in step S2.1, a table SQL is constructed, is uniformly distributed to each node, sharding logic is performed on the table SQL to obtain sharded SQL, the sharded SQL is controlled and pulled in parallel, the pulled data is format-converted to be consistent with the data format of the incremental synchronization, and final data is obtained; S2.

4. The final data of step S2.2 and the final data obtained in step S2.3 are taken as the query events; S3. The add-delete-modify-query events generated in step S2 are routed to a distributed parallel Stream pipeline; S4. The parallel Stream pipeline routes and distributes the add-delete-modify-query events to Actor entities in the Actor model sharded cluster in an asynchronous request-response mode according to the dimension fields configured for each table; S5. In the Actor entities in the Actor model sharded cluster, event persistence and state updating are performed according to the order of the add-delete-modify-query events, and complex conversion logic based on the current state is performed, and the result is returned to the Stream pipeline; The complex conversion logic based on the current state comprises: S5.

1. The current state in the Actor entities in the Actor model sharded cluster is composed of multiple sub-tables obtained by logical division of the dimension fields in step S4; S5.

2. The operations on the sub-tables in the current state are encapsulated into a SQL mode, and a temporary intermediate table is obtained after SQL mode computing processing, and is taken as an input table of the next operation; S5.

3. The operations of multiple sub-tables are combined to obtain complex conversion logic; S6. The Stream pipeline performs message batching, sorting and deduplication processing on the Response events; S7. The result of step S6 is written into the Sink data source, and an end-to-end asynchronous ACK mechanism of the Source data source and the Sink data source is performed, and if the confirmation is successful, the data synchronization is completed.

2. The data synchronization method with full link backpressure feature according to claim 1, characterized in that, In step S2, the millisecond timestamp ts of the log in the database binlog log, the file name suffix of the database binlog log, and the offset of the add, delete, modify and query event in the file of the database binlog log are combined to form the uid attribute, and the order of the add, delete, modify and query event is accurately determined based on the uid and ts information.

3. The data synchronization method with full link backpressure feature according to claim 1, characterized in that, In step S7, the end-to-end asynchronous ACK mechanism of the Source data source and the Sink data source is performed, specifically including: In step S2, the add, delete, modify and query event has the address ActorRef and the unique AckId attribute of message confirmation, and after the data is successfully written into the Sink data source, the end-to-end asynchronous ACK processing of the Source data source to the Sink data source is performed according to the ActorRef and the AckId, if the confirmation fails, the incremental synchronization based on the database binlog log will be restarted and reset to the latest successfully ACKed position to continue the recovery synchronization, and the full synchronization based on the SQL mode will continue to synchronize, and after the full synchronization ends, the unconfirmed message retry is performed.

4. A system for implementing the data synchronization method with full link back pressure characteristics according to any one of claims 1 to 3, characterized in that, Including: a task metadata module for configuring and persisting task metadata; a full data synchronization module for efficiently and in parallel synchronizing the inventory data; an incremental data synchronization module for extracting and analyzing the binlog log of the mainstream relational database; a data Sink module supporting final result persistence; a sharded Actor cluster management module for distributed management of massive Actor entities and configurable Actor computing logic; a dynamic parallelism management module for managing synchronization parallelism; a timing data inspection module for ensuring data integrity and accuracy.

Images