Data synchronization method and data synchronization device

Abstract

This invention provides a data synchronization method and apparatus. Through a visual interface, the data source and target cluster index parameters are configured. A data synchronization task is created based on the data source and index parameters. The task's time period and triggering rules are set. The task is scheduled according to these rules. An incremental watermark algorithm is used to read data from the data source. Write nodes are dynamically allocated based on load. A search interface combining HTTP / Socket dual protocols and local caching is provided to query data for business scenarios. This improves development efficiency, ensures data consistency and reliability, enhances resource utilization, and reduces data synchronization time. It significantly improves the data synchronization efficiency and system stability of distributed systems and is suitable for high-concurrency, multi-source, heterogeneous data synchronization scenarios.

Description

Technical Field

[0001] This invention relates to the field of distributed system data synchronization technology, and in particular to a data synchronization method and a data synchronization device. Background Technology

[0002] Elasticsearch (ES) is a distributed, highly scalable, and real-time search and data analysis engine that is widely used in big data fields such as finance and e-commerce.

[0003] However, enterprises currently face the following problems when using Elasticsearch for data synchronization: 1. Complex development: Traditional data synchronization requires customized code development, which has a long cycle and high maintenance costs; 2. Poor scalability: Due to the lack of unified configuration management, it is difficult to support flexible access to multiple data sources (such as MySQL, HBase, Oracle) and multiple clusters (ES, Solr); 3. Insufficient real-time performance: Incremental data synchronization depends on manual triggering, which cannot meet the high real-time business requirements; 4. Stability risks: Task scheduling and status monitoring are scattered, which can easily lead to data loss or synchronization delays.

[0004] Providing a data synchronization method that is applicable to high-concurrency, multi-source, heterogeneous data synchronization scenarios is crucial for improving data synchronization efficiency and system stability. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an efficient, stable and scalable data synchronization method and data synchronization device.

[0006] To address the aforementioned issues, this invention provides a data synchronization method, comprising: configuring index parameters for at least one data source and at least one target cluster through a visual configuration interface; creating a data synchronization task based on the data source and the index parameters, and setting the task's time period and triggering rules; scheduling the data synchronization task according to the time period and triggering rules; reading data from the data source using an incremental watermark algorithm; dynamically allocating write nodes according to the load of the target cluster and writing the data to the write nodes; and querying data for a business scenario through a search interface, wherein the search interface provides near real-time query services through HTTP / Socket dual protocols and local caching.

[0007] In some embodiments, the data source is a relational database or a non-relational database, and the target cluster is a distributed search server consisting of multiple nodes.

[0008] In some embodiments, the index parameters include an index function, which includes a timestamp segmentation function and a field mapping function; the timestamp segmentation function is used to perform timestamp segmentation on the transaction log of the data source, and the field mapping function is used to map specific fields of the data source.

[0009] In some embodiments, the task time period and the task triggering rules are stored through a distributed program coordination service, and the data synchronization task is scheduled according to the task time period and the task triggering rules.

[0010] In some embodiments, a refresh lock is used to ensure the atomicity of multi-node task execution.

[0011] In some embodiments, the data synchronization task is a full synchronization task or an incremental synchronization task; when the data source or the target cluster is abnormal, the incremental synchronization task is stopped and automatically resumed after recovery.

[0012] In some embodiments, the data synchronization task can be manually triggered or stopped through a visual execution interface.

[0013] In some embodiments, the data synchronization method further includes monitoring the data synchronization task and displaying the status of the data synchronization task through a visual monitoring interface; the status of the data synchronization task is running, stopped, or abnormal.

[0014] In some embodiments, the resources of the target cluster are extended via plug-ins.

[0015] To address the aforementioned issues, this invention also provides a data synchronization device, comprising: a configuration management module, which includes a visual configuration interface for configuring index parameters of at least one data source and at least one target cluster; a task scheduling module, which creates data synchronization tasks based on the data source and the index parameters, sets the task time period and task triggering rules for the data synchronization tasks, and schedules the data synchronization tasks based on the task time period and the task triggering rules; a data processing routing module, which includes a reading unit and a writing unit, wherein the reading unit reads data from the data source using an incremental watermark algorithm, and the writing unit dynamically allocates writing nodes based on the load of the target cluster and writes the data to the writing nodes; and a search module, which queries data for business scenarios through a search interface, wherein the search interface provides near real-time query services through HTTP / Socket dual protocols and local caching.

[0016] In some embodiments, the data synchronization device further includes a monitoring module, which is used to monitor the status of the data synchronization task and display the status of the data synchronization task through a visual monitoring interface. The status of the data synchronization task is running, stopped, or abnormal.

[0017] In some embodiments, the data processing routing module further includes a data conversion unit, which is used to perform timestamp segmentation on the transaction log of the data source using a timestamp segmentation function, and to map specific fields of the data source using a field mapping function.

[0018] The above technical solution allows for the configuration of data source and target cluster index parameters through a visual interface. Data synchronization tasks are created based on the data source and index parameters, and the task time period and triggering rules are set. These tasks are then scheduled according to the specified time period and triggering rules. An incremental watermark algorithm is used to read data from the data source, and write nodes are dynamically allocated based on load. A search interface combining HTTP / Socket dual protocols and local caching is provided to query data for business scenarios. This improves development efficiency, ensures data consistency and reliability, enhances resource utilization, and reduces data synchronization time. It significantly improves the data synchronization efficiency and system stability of distributed systems and is suitable for high-concurrency, multi-source, heterogeneous data synchronization scenarios.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a data synchronization method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a data synchronization device provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Figure 1 This is a flowchart of a data synchronization method provided in an embodiment of the present invention. Figure 1 As shown, the data synchronization method includes the following steps: Step S11: Configure the index parameters of at least one data source and at least one target cluster through the visual configuration interface.

[0024] The data source can be a relational database or a non-relational database. Relational databases include Oracle and MySQL, while non-relational databases include HBase.

[0025] The target cluster is a distributed search server composed of multiple nodes. For example, the target cluster is a multi-node Elasticsearch cluster. The roles of nodes in an Elasticsearch cluster include data nodes, master nodes, or coordinating nodes. When a node's role is that of a data node, it can be used to store data and perform search operations. In another embodiment, the target cluster is a multi-node Solr cluster.

[0026] The index parameters include a timestamp and an index suffix identifier. The index parameters also include index functions, which include a timestamp segmentation function and a field mapping function. The timestamp segmentation function is used to segment the transaction log of the data source according to timestamps; the field mapping function is used to modify the content of specific fields, including modifying the type and value of specific fields.

[0027] In this embodiment, the index function may further include a custom script, which is used to customize the mapping relationship of the index fields, reduce the complexity of index configuration, and developers can upload the custom script through the visual configuration interface as the index function.

[0028] Step S12: Create a data synchronization task based on the data source and the index parameters, and set the task time period and task triggering rules for the data synchronization task.

[0029] The data synchronization task is used to synchronize data from the data source to the target cluster. In this embodiment, each data synchronization task supports synchronizing data from multiple data sources to multiple target clusters.

[0030] The data synchronization task can be a full synchronization task or an incremental synchronization task. The task time period is the execution time window of the data synchronization task, and the task triggering rules define when and how the changes of the data source are synchronized to the target cluster.

[0031] The task triggering rules include: time-based triggering rules, such as timed triggering or triggering at specific time intervals; event-based triggering rules, such as database data change triggering; and condition-based triggering rules, such as triggering when the data volume or number of changes reaches a threshold, or triggering when the value of a specific field changes.

[0032] When the data synchronization task is created, the data source is connected with one click through a standardized interface (e.g., EnConnection) according to the data source configured in step S12.

[0033] In this embodiment, the data synchronization method is based on the River platform, and the data synchronization task is RiverJob.

[0034] The data synchronization task's time period and triggering rules are stored through a distributed program coordination service. In this embodiment, the distributed program coordination service is Zookeeper.

[0035] Step S13: Schedule the data synchronization task according to the task time period and the task triggering rules.

[0036] In this embodiment, the data synchronization task is scheduled through the distributed program coordination service Zookeeper. Zookeeper triggers the data synchronization task to perform data synchronization based on the configured index parameters and task information. The task information includes the task time period and the task triggering rules.

[0037] ZooKeeper is a distributed coordination service. A ZooKeeper cluster can store the configuration, status, and coordination information of data synchronization tasks. Task configurations can be dynamically modified via the ZooKeeper client without restarting the system. ZooKeeper can also dynamically detect index changes in the target cluster and automatically switch indexes.

[0038] The distributed program coordination service employs a distributed lock mechanism to ensure the data synchronization task. Specifically, a refresh lock is used to ensure the atomicity of multi-node task execution.

[0039] Zookeeper itself has a distributed lock mechanism, which can ensure concurrency control and data consistency in a distributed environment, avoid deadlock and race conditions, and thus guarantee high performance and reliability.

[0040] The refresh lock mechanism includes: the node that acquires the lock periodically sends refresh requests to Zookeeper to extend the lock's validity period; if a node fails or the task is completed, the lock is automatically released. Through a "unique holding + dynamic renewal" mechanism, the refresh lock provides atomicity guarantees for multi-node tasks in a distributed environment, making it particularly suitable for long-running, high-concurrency task scenarios.

[0041] This step can also be manually triggered or stopped through a visual execution interface. In this embodiment, the visual execution interface provides operations such as running a full synchronization task, running an incremental synchronization task, stopping a full synchronization task, and stopping an incremental synchronization task, which developers can manually execute as needed.

[0042] Step S14: Use the incremental water level algorithm to read data from the data source.

[0043] Each data synchronization task corresponds to at least one read stream and at least one write stream. The read stream and the write stream are processed in parallel. Different read streams are configured for different data sources. For example, if the data sources include MySQL, HBase, and Oracle, then MySQLReaderFlow, HBaseReaderFlow, and OracleReaderFlow are configured respectively to read data from different data sources, and the different read streams are processed in parallel.

[0044] The read stream uses an incremental watermark algorithm to read data from the data source. In this embodiment, the incremental watermark algorithm is based on the transaction log of the data source, extracts the timestamp information from the log, segments the transaction log according to the timestamp, and dynamically marks the "watermark" (i.e., the starting point of the incremental data), thereby accurately defining the dataset for each incremental synchronization.

[0045] In some embodiments, the data is the original data; in this embodiment, the original data is transformed to obtain the data.

[0046] In this embodiment, the step of reading data from the data source using the incremental watermark algorithm further includes: (1) using the timestamp segmentation function to perform timestamp segmentation on the transaction log of the data source; and (2) using the field mapping function to map specific fields of the data source.

[0047] In step (1), for example, if the data source is MySQL, the MySQL Binlog is retrieved based on the transaction log of the data source. The timestamp of each log entry in the Binlog is extracted. The incremental watermark algorithm segments the logs according to the timestamp (e.g., from 10:00:00 on 2025-01-01 to 10:01:00 on 2025-01-01 as one segment). The current watermark is compared with the start time of the segment. If the start time of the segment is later than the watermark, it is determined to be unsynchronized incremental data and included in the current synchronization range, thereby accurately identifying the range of incremental data and reducing redundant synchronization. After the incremental synchronization is completed, the watermark is updated to the maximum timestamp of the segment (e.g., 10:01:00 on 2025-01-01).

[0048] In step (2), the type or value of a specific field is changed through the field mapping function.

[0049] Through the processing in steps (1) and (2), the collected raw data is converted into data that conforms to the writing rules of the target cluster.

[0050] Step S15: Dynamically allocate write nodes according to the load of the target cluster and write the data to the write nodes.

[0051] This step involves collecting load information from each node in the target cluster in real time, selecting write nodes that meet the criteria, and distributing the processed data to these write nodes. For example, the node with the lowest load is selected as the write node.

[0052] This step improves resource utilization by dynamically allocating the write nodes, thereby reducing hardware investment and saving costs.

[0053] The write stream of the data synchronization task writes data to the write node. The write stream is configured separately for different target clusters. For example, if the target cluster includes Elasticsearch and Solr, then ElasticsearchWriterFlow and SolrWriterFlow are configured respectively to write the data to different target clusters.

[0054] The different write streams are processed in parallel, and the write streams are processed in parallel with the read streams.

[0055] Step S16: Query data for the business scenario through the search interface. The search interface provides near real-time query service through HTTP / Socket dual protocol and local caching.

[0056] In this embodiment, the business scenarios include order, membership, marketing and other scenarios. The local cache caches high-frequency query results and combines index functions to achieve near real-time data updates, which can reduce data synchronization latency to the second level.

[0057] The above technical solution improves development efficiency by enabling data source access, index mapping, and task scheduling through a visual interface; ensures data consistency and reliability through the unified distributed program coordination service Zookeeper; improves resource utilization by dynamically allocating write nodes based on load; reduces data synchronization time through the incremental watermark algorithm; and provides near real-time query services through HTTP / Socket dual protocols and local caching.

[0058] In this embodiment, the data synchronization method further includes monitoring the data synchronization task and displaying the status of the data synchronization task through a visual monitoring interface.

[0059] The data synchronization task is in a running state, a stopped state, or an abnormal state. In this embodiment, the visual monitoring interface includes a running status display area, which displays the status of each data synchronization task.

[0060] The monitoring of the data synchronization tasks also includes monitoring the synchronization progress and resource usage of the data synchronization tasks. In this embodiment, the visual monitoring interface includes a storage status display area and a thread status display area. The storage status display area displays the storage resource usage of each data synchronization task, and the thread status display area displays the stage of each data synchronization task's thread.

[0061] When the data source or the target cluster malfunctions, the incremental synchronization task is stopped, and automatically resumed upon recovery to avoid data corruption or system overload. When an interruption is triggered, the last timestamp of the incremental synchronization and the identifier of the processed data are saved; when synchronization resumes, the unsynchronized data is read and processed starting from the position corresponding to the last timestamp.

[0062] In this embodiment, the resources of the target cluster can also be extended through plugins. For example, Kafka and Flink can be connected to the target cluster through plugins.

[0063] Based on the same inventive concept, the present invention also provides a data synchronization device capable of executing the data synchronization method described above.

[0064] like Figure 2 As shown, the data synchronization device includes: a configuration management module 21, a task scheduling module 22, a data processing routing module 23, and a search module 24.

[0065] The configuration management module 21 includes a visual configuration interface, which is used to configure index parameters for at least one data source and at least one target cluster. Adding a data source through the visual configuration interface can significantly shorten configuration time and supports more than 10 heterogeneous databases.

[0066] The task scheduling module 22 is used to create a data synchronization task based on the data source and the index parameters, set the task time period and task triggering rules of the data synchronization task, and schedule the data synchronization task based on the task time period and task triggering rules.

[0067] The data synchronization task includes a full synchronization task and an incremental synchronization task.

[0068] The task scheduling module 22 schedules the data synchronization task through a distributed program coordination service. The task time period and the task triggering rules can be stored in the distributed program coordination service.

[0069] The task scheduling module 22 ensures the data synchronization task through the distributed lock mechanism of the distributed program coordination service, including: using refresh locks to ensure the atomicity of multi-node task execution, thereby ensuring data consistency and reliability.

[0070] In this embodiment, the distributed program coordination service is Zookeeper.

[0071] The task scheduling module 22 also includes a visual execution interface, which provides operations such as running a full synchronization task, running an incremental synchronization task, stopping a full synchronization task, and stopping an incremental synchronization task. Researchers can manually perform these operations as needed.

[0072] The data processing routing module 23 includes a reading unit 231 and a writing unit 232. The reading unit 231 is used to read data from the data source using an incremental watermark algorithm. The writing unit 232 is used to dynamically allocate writing nodes according to the load of the target cluster and write the data to the writing nodes. Throughput is improved by dynamically allocating writing nodes.

[0073] In this embodiment, the data processing routing module 23 further includes a data conversion unit 233, which converts the original data into the data.

[0074] In this embodiment, the reading unit 231 uses an incremental watermark algorithm to read data from the data source. The data conversion unit 233 uses a timestamp segmentation function to segment the transaction log of the data source into timestamps, and uses a field mapping function to map specific fields of the data source, changing the type or value of the specific fields.

[0075] The search module 24 includes a search interface, which implements near real-time query service through HTTP / Socket dual protocol and local cache. The search module 24 is used to provide near real-time query service through the search interface, which greatly reduces the latency of the interface.

[0076] The data synchronization device further includes a monitoring module 25, which includes a visual monitoring interface. The monitoring module 25 is used to monitor the data synchronization task and display the status of the data synchronization task on the visual monitoring interface.

[0077] When the monitoring module 25 detects an anomaly in the data source or the target cluster, it triggers a stop to the incremental synchronization task. Once the data is restored, it will automatically resume transmission, further ensuring data consistency and reliability.

[0078] The above technical solutions significantly improve the data synchronization efficiency and system stability of distributed systems, and are suitable for high-concurrency, multi-source heterogeneous data synchronization scenarios.

[0079] It should be noted that references to "an embodiment," "an embodiment," "an exemplary embodiment," "some embodiments," etc., in the specification indicate that the described embodiments may include specific features, structures, or characteristics, but each embodiment may not necessarily include that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. In addition, when a specific feature, structure, or characteristic is described in connection with an embodiment, whether explicitly described or not, implementing such a feature, structure, or characteristic in conjunction with other embodiments is within the knowledge of those skilled in the art.

[0080] It should be noted that the terms "comprising" and "having," and their variations, used in this invention document are intended to cover non-exclusive inclusion. The terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence, unless explicitly indicated by the context. It should be understood that such data used interchangeably where appropriate. Furthermore, embodiments and features within embodiments of this invention can be combined with each other unless otherwise specified. In addition, descriptions of well-known components and technologies have been omitted in the above description to avoid unnecessarily obscuring the concepts of this invention. In the various embodiments described above, each embodiment focuses on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0081] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data synchronization method, characterized in that, include: Configure index parameters for at least one data source and at least one target cluster through a visual configuration interface; A data synchronization task is created based on the data source and the index parameters, and the task time period and triggering rules of the data synchronization task are set. The data synchronization task is scheduled according to the task time period and the task triggering rules; Data is read from the data source using an incremental water level algorithm; Dynamically allocate write nodes according to the load of the target cluster and write the data to the write nodes; Data for business scenarios can be queried through a search interface, which provides near real-time query services via HTTP / Socket dual protocols and local caching.

2. The data synchronization method according to claim 1, characterized in that, The data source is a relational database or a non-relational database, and the target cluster is a distributed search server composed of multiple nodes; wherein, the resources of the target cluster are extended through plug-ins.

3. The data synchronization method according to claim 1, characterized in that, The index parameters include index functions, which include timestamp segmentation functions and field mapping functions; The incremental watermark algorithm uses the timestamp segmentation function to segment the transaction logs of the data source using timestamps, and uses the field mapping function to map specific fields of the data source.

4. The data synchronization method according to claim 1, characterized in that, The data synchronization task can be manually triggered or stopped through the visual execution interface; The distributed program coordination service stores the task time period and the task triggering rules, and schedules the data synchronization task according to the task time period and the task triggering rules.

5. The data synchronization method according to claim 1, characterized in that, A refresh lock is used to ensure the atomicity of multi-node task execution.

6. The data synchronization method according to claim 1, characterized in that, The data synchronization task can be a full synchronization task or an incremental synchronization task; when the data source or the target cluster is abnormal, the incremental synchronization task is stopped and will automatically resume after recovery.

7. The data synchronization method according to claim 1, characterized in that, The data synchronization method further includes monitoring the data synchronization task and displaying the status of the data synchronization task through a visual monitoring interface. The status of the data synchronization task is either running, stopped, or abnormal.

8. A data synchronization device, characterized in that, include: A configuration management module, comprising a visual configuration interface, is used to configure index parameters for at least one data source and at least one target cluster through the visual configuration interface. The task scheduling module is used to create a data synchronization task based on the data source and the index parameters, set the task time period and task triggering rules of the data synchronization task, and schedule the data synchronization task based on the task time period and the task triggering rules. The data processing routing module includes a reading unit and a writing unit. The reading unit is used to read data from the data source using an incremental watermark algorithm, and the writing unit is used to dynamically allocate writing nodes according to the load of the target cluster and write the data to the writing nodes. The search module is used to query data in business scenarios through a search interface, which provides near real-time query services through HTTP / Socket dual protocols and local caching.

9. The data synchronization device according to claim 8, characterized in that, It also includes a monitoring module, which is used to monitor the status of the data synchronization task and display the status of the data synchronization task through a visual monitoring interface. The status of the data synchronization task is running, stopped, or abnormal.

10. The data synchronization device according to claim 8, characterized in that, The data processing routing module further includes a data conversion unit, which is used to perform timestamp segmentation on the transaction log of the data source using a timestamp segmentation function, and to map specific fields of the data source using a field mapping function.

Images