Data synchronization method and device, electronic equipment and storage medium

Abstract

The invention discloses a data synchronization method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a rule engine and a business processing rule according to the structured data of a migration source database; generating a data processing task based on the rule engine, the business processing rule and the structured data; scheduling the data processing task to a task processing server to process the structured data; and acquiring change data of the structured data through a change service capture server, and performing data synchronization on the structured data in a migration target database based on the change data. According to the embodiment of the invention, the problem of data synchronization among multiple storage systems can be solved, the data synchronization efficiency can be improved, the real-time performance of data synchronization can be enhanced, the consistency of data storage can be ensured, and the security of data storage can be enhanced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data synchronization method, device, electronic device and storage medium. Background Art

[0002] With the advent of the big data era, enterprises have an increasing demand for data processing, especially for the application of distributed databases. Take TiDB as an example. As a high-performance distributed relational database, it can support hybrid transactional and analytical processing (HTAP), which can meet the dual needs of online transaction processing and analysis. Then how to achieve efficient and real-time data synchronization between multiple systems has become a major challenge. Therefore, how to synchronize data between different systems to ensure data consistency and real-time performance has become a problem that needs to be solved urgently. Summary of the invention

[0003] The present invention provides a data synchronization method, device, electronic device and storage medium to solve the data synchronization problem between multiple storage systems, which can improve data synchronization efficiency, enhance the real-time performance of data synchronization, ensure the consistency of data storage, and enhance the security of data storage.

[0004] According to one aspect of the present invention, a data synchronization method is provided, wherein the method comprises:

[0005] Obtain the rule engine and business processing rules based on the structured data of the migration source database;

[0006] Generate a data processing task based on the rule engine, the business processing rule and the structured data;

[0007] Dispatching the data processing task to a task processing server to process the structured data;

[0008] The change data of the structured data is acquired by the change service capture server, and the structured data is synchronized in the migration target database based on the change data.

[0009] According to another aspect of the present invention, a data synchronization device is provided, wherein the device comprises:

[0010] The processing preparation module is used to obtain the rule engine and business processing rules based on the structured data of the migration source database;

[0011] A task construction module, used to generate a data processing task based on the rule engine, the business processing rules and the structured data;

[0012] A task scheduling module, used for scheduling the data processing task to a task processing server to process the structured data;

[0013] The change synchronization module is used to obtain the change data of the structured data through the change service capture server, and synchronize the structured data in the migration target database based on the change data.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data synchronization method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data synchronization method described in any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention determines the rule engine and business processing rules for the structured data of the migration source database, generates data processing tasks for the rule engine, business processing rules and structured data, and schedules the data processing tasks to the task processing server, so that the task processing server processes the structured data, obtains the change data of the structured data through the change service capture server, and synchronizes the structured data in the migration target database through the change data, thereby realizing data synchronization between different storage databases and ensuring data consistency between different systems. By actively capturing the change data through the change service capture server, the waiting time for data synchronization can be reduced, the data synchronization efficiency can be improved, the real-time performance of data synchronization can be enhanced, the consistency of data storage can be ensured, and the security of data storage can be enhanced.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of a data synchronization method provided according to Embodiment 1 of the present invention;

[0023] Figure 2 is a flowchart of another data synchronization method provided according to Embodiment 2 of the present invention;

[0024] Figure 3 is a flowchart of another data synchronization method provided according to Embodiment 3 of the present invention;

[0025] Figure 4 is a flowchart of another data synchronization method provided according to Embodiment 4 of the present invention;

[0026] Figure 5 This is an example diagram of the architecture of CDC monitoring in a data synchronization process provided in Embodiment 5 of the present invention;

[0027] Figure 6 This is a diagram of a message transmission example of a Kafka architecture provided according to Embodiment 5 of the present invention;

[0028] Figure 7 This is an example diagram of an architecture for data processing in a data synchronization process provided according to Embodiment 5 of the present invention;

[0029] Figure 8 is a structural diagram of a data synchronization device provided according to Embodiment 6 of the present invention;

[0030] Fig. 9 It is a structural schematic diagram of an electronic device for implementing the data synchronization method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment 1

[0034] Figure 1 1 is a flow chart of a data synchronization method provided according to Embodiment 1 of the present invention. This embodiment is applicable to data synchronization between different storage systems. The method can be executed by a data synchronization device, which can be implemented in the form of hardware and / or software. Figure 1 As shown, the method includes:

[0035] Step 110: Obtain a rule engine and business processing rules according to the structured data of the migration source database.

[0036] Among them, the migration source database can be a database generated among multiple storage databases that needs to migrate and synchronize data. The migration source database can contain one or more structured data. The structured data can be data with a clear format and structure. The structured data can be stored in the migration source database.

[0037] In the embodiment of the present invention, the business processing rules may be rules for operating and processing structured data, and the business processing rules may be pre-configured or input by the user in real time according to business requirements. The business processing rules may include rules for operations such as data update, data deletion, and data insertion, and the business processing rules may be implemented through database structured query statements or other scripting languages.

[0038] Specifically, a corresponding rule engine and business processing rules can be configured for the structured data in the migration source database. The migration source database can be one or more of a plurality of different storage databases. For example, a plurality of different storage databases can be used as migration source databases respectively, and the structured data in the plurality of migration source databases can be allocated with pre-configured corresponding rule engines and business processing rules. It can be understood that the configured business processing rules can be used to process part or all of the structured data stored in the migration source database. Alternatively, a rule engine and business processing rules can be selected for the structured data of the migration source database according to user needs. For example, a user can determine a migration source database in one or more storage databases according to their own business needs, and can determine part or all of the structured data in the migration source database as structured data to be processed. A rule engine can be used for these structured data, and corresponding business processing rules can be written according to the rule engine.

[0039] Step 120: Generate a data processing task based on the rule engine, business processing rules and structured data.

[0040] The data processing task may be a scheduling task for processing structured data, and the data processing task may be generated by a rule engine, business processing rules, and structured data. The encapsulation form of the data processing task may be determined by a communication protocol form, and the communication protocol may be pre-configured or automatically selected. The data processing task may be transmitted through an API call, a JDBC connection, etc. The data processing task may include indication information of information such as a rule engine, business processing rules, and structured data, and the indication information may include numbers or names of the rule engine, business processing rules, and structured data, etc.

[0041] In an embodiment of the present invention, information such as rule engine, business processing rules and structured data can be encapsulated as a data processing task. For example, numbering information of rule engine, business processing rules and structured data can be extracted, and the numbering information can be encapsulated as a data processing task.

[0042] Step 130: Schedule the data processing task to the task processing server to process the structured data.

[0043] The task processing server may be a server or a server cluster that processes data processing tasks, the task processing server may receive data processing tasks, and the task processing server may be pre-connected to a task management center that generates data processing tasks.

[0044] In an embodiment of the present invention, data processing tasks can be assigned to task processing servers through specified rules, so that the task processing server determines information such as the rule engine, business processing rules, and structured data according to the data processing tasks, thereby processing the structured data with the help of the rule engine and the business processing rules, wherein the specified rules can be pre-configured rules for assigning data processing tasks, and the specified rules can include allocating data processing tasks according to parameters such as server performance or server attributes of different task processing servers.

[0045] Step 140: Acquire the change data of the structured data through the change service capture server, and synchronize the structured data in the migration target database based on the change data.

[0046] Among them, the change service capture server (Change Data Capture Server) can be a system that captures and tracks data changes in the database, which has functions such as real-time data synchronization, data integration, and business process automation. The change service capture server can be more service capture servers usually establish a connection with the database and listen to the database's transaction log or other data change notification mechanism. When the data in the database changes, the server will immediately detect these changes. The change service capture server may include TiCDC, Debezium, Maxwell, and Flink CDC, etc. The change data may be a data record indicating the structured data that has changed in the database. The change data may include inserting data, updating data, and deleting data, etc. The change data may exist in the form of a database transaction log. The migration target database may be a target database for synchronous migration of structured data, and data synchronization may be a process that indicates the structured data that has changed in the migration source database, and corresponding changes can be made in the migration target database.

[0047] In an embodiment of the present invention, a change service capture server can be pre-configured, and the change service capture server can be used to obtain change data corresponding to the structured data. The change data can be transmitted to the migration target database, so that the migration target database makes corresponding data changes to the structured data according to the change data, thereby realizing data synchronization of the structured data in the migration source database and the migration target database.

[0048] The embodiment of the present invention determines a rule engine and business processing rules for the structured data of the migration source database, generates a data processing task for the rule engine, business processing rules and structured data, and schedules the data processing task to a task processing server, so that the task processing server processes the structured data, obtains the change data of the structured data through a change service capture server, and synchronizes the structured data in the migration target database through the change data, thereby realizing data synchronization between different storage databases and ensuring data consistency between different systems. By actively capturing the change data through the change service capture server, the waiting time for data synchronization can be reduced, the efficiency of data synchronization can be improved, the real-time performance of data synchronization can be enhanced, the consistency of data storage can be ensured, and the security of data storage can be enhanced.

[0049] Embodiment 2

[0050] Figure 2 is a flowchart of another data synchronization method provided according to the second embodiment of the present invention. The implementation of the present invention is a specific implementation based on the above-mentioned embodiment of the invention, and describes the process of determining the rule engine and the business processing rules. Figure 2 The method provided in the embodiment of the present invention specifically includes the following steps:

[0051] Step 210: determine a migration source database in a preset database cluster, and select a structured data table in the migration source database as structured data.

[0052] Among them, the preset database cluster can be a cluster composed of multiple storage databases, the system architectures between the multiple storage databases included in the preset database cluster can be the same or synchronized, the structured data table can be a data table storing structured data in the migration source database, and the structured data table can store corresponding structured data for different businesses.

[0053] In an embodiment of the present invention, one or more storage databases can be determined as migration source databases in a preset database cluster according to user needs or preset configurations. For each migration source database, one or more structured data tables can be selected in the corresponding migration source database, and the data stored in the selected structured data tables can be used as structured data.

[0054] Step 220: Obtain rule engine selection information and business rule writing information input by the user.

[0055] The engine selection information may be information for selecting a rule engine, and the engine selection information may be triggered in different forms, such as user input through a visual interface or voice input, and the engine selection information may at least include information indicating the selected rule engine, such as the name of the rule engine or the number of the rule engine, etc. The business rule writing information may be information for generating business processing rules, and the business rule writing information may be input externally, and the business rule writing information may be implemented in different programming languages, such as SQL language or R language, etc.

[0056] In an embodiment of the present invention, a user may input rule engine search information and business rule writing information through a visual interface or language. For example, a visual interface may be provided to the user, and the visual interface may display one or more rule engines to be selected. The user may trigger rule engine selection information by selecting a rule engine in the visual interface. The rule engine selection information may at least include an identifier of the rule engine selected by the user in the visual interface. Furthermore, after obtaining the rule engine selection information, a business rule writing page may be provided in accordance with the rule execution syntax of the rule engine. The user may write business rules in the business rule page, and the information input by the user in the writing page may be used as business rule writing information. It can be understood that the business rule writing page may correspond to the rule engine, that is, rule engines of different types or different business processing rules may have different writing pages.

[0057] Step 230: Use the preset rule engine corresponding to the rule engine selection information as the rule engine, and use the business rule writing information as the business processing rule.

[0058] Specifically, one or more corresponding preset rule engines can be searched according to the rule engine selection information input by the user, that is, the identification information of the preset rule engine can be included in the rule selection information input by the user, and the received business rule writing information can be processed into business processing rules. For example, the business rule writing information can be saved in a specified format, and the saved file can be used as a business processing rule.

[0059] Step 240: Generate a data processing task based on the rule engine, business processing rules and structured data.

[0060] Step 250: dispatching a data processing task to a task processing server to process structured data;

[0061] Step 260: Acquire the change data of the structured data through the change service capture server, and synchronize the structured data in the migration target database based on the change data.

[0062] According to an embodiment of the present invention, a migration source database is selected in a preset database cluster, and a structured data table is selected in the migration source database to obtain structured data. For the structured data, rule engine selection information and business rule writing information input by the user can be obtained, and a preset rule engine corresponding to the rule engine selection information is determined as the rule engine, and the business rule writing information is used as the business processing rule, and a data processing task is generated by the rule engine, the business processing rule and the structured data, and the data processing task is scheduled to a task processing server, so that the task processing server processes the structured data, and obtains the change data of the structured data through the change service capture server, and the structured data is synchronously processed in the migration target database through the change data, so that data synchronization between different storage databases can be achieved, and data consistency between different systems can be guaranteed. By actively capturing the change data through the change service capture server, the waiting time for data synchronization can be reduced, the data synchronization efficiency can be improved, the real-time performance of data synchronization can be enhanced, the consistency of data storage can be ensured, and the security of data storage can be enhanced.

[0063] Embodiment 3

[0064] Figure 3 is a flowchart of another data synchronization method provided according to Embodiment 3 of the present invention. The embodiment of the present invention describes the allocation method of data processing tasks based on the above-mentioned embodiment of the invention. Figure 3 The method provided in the embodiment of the present invention specifically includes the following steps:

[0065] Step 310: Obtain a rule engine and business processing rules according to the structured data of the migration source database.

[0066] Step 320: Generate a data processing task based on the rule engine, business processing rules and structured data.

[0067] Step 330: Determine the server load of different task processing servers.

[0068] The server load may be the business processing pressure and resource usage of the task processing server during operation. The server load may include CPU load, memory load, disk load, network load, etc.

[0069] In an embodiment of the invention, the server loads of all task processing servers may be monitored, thereby obtaining server loads such as CPU load, memory load, disk load, and network load.

[0070] Step 340: distribute the data processing tasks to the task queue of the task processing server according to the server load, so that the task processing server processes the data processing tasks according to the order of the task queue.

[0071] Among them, the task queue can be a processing queue for caching data processing tasks in the task processing server. The data processing tasks in the task queue can follow the first-in-first-out processing rule, that is, the data processing tasks that enter the task queue first are processed first.

[0072] In an embodiment of the present invention, the server loads of the task servers may be compared, and the data processing task may be assigned to a matching task processing server according to the comparison result. The matching may include that the server load can support the operation of the data processing task.

[0073] Step 350: Acquire the change data of the structured data through the change service capture server, and synchronize the structured data in the migration target database based on the change data.

[0074] The embodiment of the present invention determines a rule engine and business processing rules for the structured data of the migration source database, generates a data processing task for the rule engine, business processing rules and structured data, obtains the server load of the task processing server, and schedules the data processing task to the task processing server through the server load, so that the task processing server processes the structured data, obtains the change data of the structured data through the change service capture server, and synchronizes the structured data in the migration target database through the change data, thereby realizing data synchronization between different storage databases and ensuring data consistency between different systems. By actively capturing the change data through the change service capture server, the waiting time for data synchronization can be reduced, the data synchronization efficiency can be improved, the real-time performance of data synchronization can be enhanced, the consistency of data storage can be ensured, and the security of data storage can be enhanced.

[0075] Further, based on the above-mentioned embodiment of the invention, the task processing server processes the data processing tasks according to the order of the task queue, including:

[0076] Perform rule verification on the business processing rules of the data processing task through the task processing server;

[0077] After verification, the rule engine that controls the task processing server to load the data processing task;

[0078] The rule engine of the task processing server processes the structured data of the data processing task according to the business processing rules.

[0079] Specifically, rule verification can be a process of verifying business processing rules. Rule verification can verify the syntax and legality of business processing rules.

[0080] In an embodiment of the present invention, the task processing server can read the data processing tasks in the task queue in sequence, can extract business processing rules, rule engines, and structured data according to the data processing tasks, and can perform rule verification on syntax and legality etc. for the extracted business processing rules. After the business processing rules pass the verification, the task processing server can load the rule engine and parse the business processing rules through the loaded rule engine, so that the task processing server processes the structured data according to the business processing rules in the rule engine, thereby realizing the change of the structured data.

[0081] Embodiment 4

[0082] Figure 4 is a flowchart of another data synchronization method provided according to Embodiment 4 of the present invention. The embodiment of the present invention is a specific embodiment based on the above embodiment, and introduces the process of synchronizing structured data. Figure 4 The data synchronization method provided by the embodiment of the present invention specifically includes the following steps:

[0083] Step 410: Obtain a rule engine and business processing rules based on the structured data of the migration source database.

[0084] Step 420: Generate a data processing task based on the rule engine, business processing rules and structured data.

[0085] Step 430: Schedule the data processing task to the task processing server to process the structured data.

[0086] Step 440: Use the change service capture server to read the transaction log of the migration source database, and extract the change data corresponding to the structured data from the transaction log, wherein the change data includes operation information and structured data table information of the structured data.

[0087] Among them, the transaction log can be a file used to record transactions in the database. The transaction log can be continuously recorded and generated by the migration database during the operation process. The transaction log may include log records. The log records may include the start time, end time, operation type, operation object, data value before the operation, and data value after the operation of the transaction. The change data can be saved in the transaction log in the form of log records. The operation information may include the start time, end time, operation type, etc. of the transaction. The structured data table information may include the operation object, data value before the operation, and data value after the operation.

[0088] In an embodiment of the present invention, a change service capture server may be pre-configured, and the transaction log of the migration source database may be monitored by the change service capture server. The change service capture server establishes a connection with the migration source database and monitors the transaction log of the migration source database. Furthermore, the change service capture server may monitor the change data of structured data through other information. For example, the change service capture server may monitor the trigger or other change notification mechanism of the migration source database. The other change notification mechanism may be implemented in different ways in different database systems, such as MySQL binary log, Oracle LogMiner, etc. After obtaining the transaction log, the change service capture server may extract the change data of the corresponding structured data in the transaction log, that is, the log record may be extracted as the change data in the transaction log.

[0089] Step 450: convert the structured data according to the data format of the migration target database by changing the service capture server.

[0090] The data format may be the organization method and relationship of data in different databases. The data format may determine the storage, organization and management method of each migration target database. Different types of migration target databases may have different data formats.

[0091] Specifically, the data format of the migration target database can be determined, and the structured data corresponding to the change data can be converted according to the data format of the migration target database by changing the service capture server, so that the structured data meets the data format of the migration target database. The data conversion can include data type conversion, field mapping, data encoding conversion, etc.

[0092] Step 460: Transmit the structured data and the changed data to the migration target database, so that the migration target database synchronizes the structured data according to the changed data.

[0093] In the embodiment of the present invention, the changed data and the structured data after data conversion can be transmitted to the migration target database, and the migration target database can process the above structured data according to the changed data, thereby achieving synchronization.

[0094] Further, based on the above-mentioned embodiment of the invention, the structured data and the changed data are transmitted to the migration target database, so that the migration target database performs synchronization processing on the structured data according to the changed data, including:

[0095] Upload structured data and changed data as subscription topics to the Kafka message queue cluster; distribute structured data and changed data to the migration target database that subscribes to the subscription topic through the Kafka cluster message queue cluster; use the migration target database to synchronize structured data according to the changed data.

[0096] Among them, the Kafka message queue cluster can be used to implement the KAFKA distributed message system architecture. The KAFKA distributed message system architecture can have the characteristics of high throughput, scalability and high availability. The KAFKA distributed message system architecture can include producers, consumers, brokers, topics, partitions, etc. The Kafka message queue cluster can be used as a cluster composed of broker components of the KAFKA distributed message system architecture. The Kafka message queue cluster can include one or more subscription topics (Topics) subscribed by the migration target database. The migration target database in the KAFKA distributed message system architecture can be used as a consumer.

[0097] In an embodiment of the present invention, the change service capture server can act as a producer of the KAFKA distributed messaging system architecture, and can upload structured data and change data as subscription topics to the Kafka message queue cluster. The Kafka message queue cluster can save subscription topics pre-configured by different consumers. The Kafka message queue cluster can cache the received subscription topics in message queues corresponding to different consumers. The Kafka message queue cluster can send the subscription topics to the migration target database as a consumer through the message queue, so that the migration target database obtains the structured data and change data in the subscription topic. The migration target database can process the structured data according to the change data, thereby realizing data synchronization between different databases.

[0098] Further, based on the above-mentioned embodiment of the invention, the structured data and the changed data are transmitted to the migration target database, and the structured data and the changed data are encrypted.

[0099] In the embodiment of the present invention, encryption technology can be used to encrypt structured data and changed data during data transmission, which can improve the security of sensitive channels.

[0100] Embodiment 5

[0101] In this embodiment of the present invention, data migration synchronization is implemented through TiDB's CDC service. In this embodiment of the present invention, the data migration synchronization process may include the following steps:

[0102] 1. Environment preparation and basic configuration

[0103] 1.1. TiDB cluster deployment: Ensure that a stable TiDB cluster has been installed and configured. The TiDB cluster can include core components such as TiDB Server, PD Server, and TiKV Server. Start the TiDB CDC service (TiCDC) and ensure that it is properly integrated with the TiDB cluster and can monitor and capture database change events. Figure 5 ,TiDB Server is a key component in the TiDB distributed database cluster, mainly responsible for processing SQL requests and interacting with clients.

[0104] PD Server is mainly responsible for the metadata management and scheduling of the cluster. Metadata management is used to store and manage metadata information of the entire TiDB cluster, including the database topology, table structure, partition information, TiKV node status, etc. Scheduling management includes: responsible for data distribution and load balancing between TiKV nodes, determining data migration and redistribution based on factors such as the storage capacity and load conditions of each node to ensure uniform distribution of data in the cluster and improve system performance and reliability. Manage the placement of TiKV node replicas to ensure that each data partition has enough replicas and that the replicas are distributed on different physical nodes to prevent single point failures and improve data availability. Handle fault detection and recovery of TiKV nodes. When a node fails, PD will automatically trigger the re-election of replicas and data migration.

[0105] TiKV Server is an important component in the TiDB cluster and is responsible for storing data. TiKV Server has a distributed architecture that stores data on multiple nodes, achieving horizontal expansion. This means that more data and requests can be processed by adding nodes, improving the performance and capacity of the system.

[0106] 1.2. Java development environment configuration: Set up the Java development environment and reference necessary dependent libraries, including but not limited to Apache Kafka client, JSON processing library (such as Jackson or Gson), and HTTP client library (such as OkHttp).

[0107] 2. Data change capture and transmission

[0108] 2.1. Configure the data change output target to Kafka or other message queue services in the TiCDC console or through the command line to ensure that change events can be pushed to the middleware in a timely and accurate manner; set the database table and change type (INSERT, UPDATE, DELETE) to be captured.

[0109] 2.2 Java application design. The Java application includes a data receiving module and a data parsing module. The data receiving module can subscribe to the change event topic published by TiCDC through the Kafka consumer client, so as to pull the data change records in the message queue in real time. The data parsing model is used to parse the received change time data and convert it into a unified data model or object. The conversion process can be achieved through operations such as JSON deserialization. Figure 6 , the workflow of the Kafka consumer client may involve ConsumerCoordinator, Fetcher, ConsumerNetworkClient, etc. ConsumerCoordinator is responsible for coordinating the operations of the consumer group, such as managing consumer group membership and allocating partitions. Fetcher is used to obtain messages from the Kafka server. ConsumerNetw orkClient is responsible for sending network requests. Its workflow specifically includes: the consumer starts pulling messages by calling the poll() method. If there is no metadata or the metadata is expired, call the ensureFreshMetadata() method to update the metadata. Then call the poll() method again. Then call the sendFetc hes() method to prepare to send a pull message request. The send() method actually sends the pull message request. After sending the network request to pull the message, the response is asynchronously written to the cache completedFetches. Get the message from the cache through the fetchedRecords() method. Finally, return the message to the caller. Thus, the acquisition process of the subscribed topic is realized.

[0110] 3. Data processing and conversion

[0111] 3.1. Design data processing logic according to business requirements, which may include data cleaning, format conversion, field mapping, business rule verification, etc. It may also include data filtering logic, which removes change records that do not need to be synchronized or routes them to different databases for processing according to specific conditions.

[0112] 3.2 Concurrency and Error Handling

[0113] Use Java's concurrency features (such as thread pools, CompletableFuture, etc.) to handle high-concurrency scenarios and ensure efficient and stable data processing. Implement retry mechanisms and exception capture, and automatically retry in the event of network instability or temporary unreachability of the target system.

[0114] 4. Target system data migration

[0115] 4.1 Data Push

[0116] See also Figure 7 , according to the interface specification or database structure of the target system, design the data push module, and write the processed data to the target system through API calls, JDBC connections or other methods. For database targets, JDBC can be used to directly execute SQL insert, update, and delete statements; for other types of targets (such as data warehouses and message queues), the corresponding interface call logic needs to be implemented according to the specific situation.

[0117] 4.2 Status Tracking and Monitoring

[0118] Implement a logging mechanism to record the processing status of each piece of data in detail (success, failure, number of retries, etc.) to facilitate problem tracking and performance monitoring. You can also develop or integrate a visual monitoring interface that displays the real-time progress, success rate, latency indicators, etc. of data migration, providing an intuitive monitoring tool for operation and maintenance personnel.

[0119] 5. Security and stability

[0120] The data transmission process is encrypted to ensure data security. Resource monitoring and automatic expansion and contraction strategies can also be implemented to ensure stable operation of the system under high load. Regular backup and recovery plans are implemented to ensure data integrity during data migration.

[0121] Embodiment 6

[0122] Figure 8 1 is a schematic diagram of the structure of a data synchronization device provided according to Embodiment 6 of the present invention. Figure 8 As shown, the device comprises:

[0123] The processing preparation module 510 is used to obtain a rule engine and business processing rules according to the structured data of the migration source database.

[0124] The task construction module 520 is used to generate data processing tasks based on the rule engine, business processing rules and structured data.

[0125] The task scheduling module 530 is used to schedule data processing tasks to the task processing server to process structured data.

[0126] The change synchronization module 540 is used to obtain the change data of the structured data through the change service capture server, and synchronize the structured data in the migration target database based on the change data.

[0127] In the embodiment of the present invention, a processing preparation module determines a rule engine and business processing rules for structured data of a migration source database, a task construction module generates a data processing task for the rule engine, business processing rules and structured data, a task scheduling module schedules the data processing task to a task processing server, so that the task processing server processes structured data, and a change synchronization module obtains change data of structured data through a change service capture server. In the embodiment of the present invention, structured data is synchronously processed in a migration target database through change data, data synchronization between different storage databases can be achieved, data consistency between different systems can be guaranteed, and change service capture servers actively capture change data, which can reduce the waiting time for data synchronization, improve data synchronization efficiency, enhance the real-time performance of data synchronization, ensure the consistency of data storage, and enhance the security of data storage.

[0128] In some inventive embodiments, the process preparation module 510 includes:

[0129] The data determination unit is used to determine a migration source database in a preset database cluster, and select a structured data table in the migration source database as structured data.

[0130] The information acquisition unit is used to acquire the rule engine selection information and business rule writing information input by the user.

[0131] The rule determination unit is used to use the preset rule engine corresponding to the rule engine selection information as the rule engine, and use the business rule writing information as the business processing rule.

[0132] In some other inventive embodiments, the task scheduling module 530 includes:

[0133] A load detection unit is used to determine the server load of different task processing servers.

[0134] The task allocation unit is used to allocate the data processing tasks to the task queue of the task processing server according to the server load, so that the task processing server processes the data processing tasks according to the order of the task queue.

[0135] In some embodiments of the invention, the task allocation unit enables the task processing server to process the data processing tasks according to the order of the task queue, including:

[0136] Perform rule verification on the business processing rules of the data processing task through the task processing server;

[0137] After verification, the rule engine that controls the task processing server to load the data processing task;

[0138] The rule engine of the task processing server processes the structured data of the data processing task according to the business processing rules.

[0139] In some inventive embodiments, the change synchronization module 540 includes:

[0140] The change data unit is used to read the transaction log of the migration source database by using the change service capture server, and extract the change data corresponding to the structured data in the transaction log, wherein the change data includes operation information and structured data table information of the structured data.

[0141] The data conversion unit is used to convert the structured data according to the data format of the migration target database by changing the service capture server.

[0142] The synchronous transmission unit is used to transmit the structured data and the changed data to the migration target database, so that the migration target database performs synchronous processing on the structured data according to the changed data.

[0143] In some embodiments of the invention, the synchronous transmission unit is specifically used to: upload structured data and change data as subscription topics to a Kafka message queue cluster; distribute the structured data and change data to a migration target database that subscribes to the subscription topic through the Kafka cluster message queue cluster; and use the migration target database to synchronize the structured data according to the change data.

[0144] In some embodiments of the invention, the synchronous transmission unit is also used to: encrypt structured data and changed data.

[0145] The data synchronization device provided in the embodiment of the present invention can execute the data synchronization method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0146] Embodiment 7

[0147] Fig. 9 Schematic diagram of the structure of an electronic device that implements the data synchronization method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0148] like Fig. 9As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0149] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0150] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a data synchronization method.

[0151] In some embodiments, the data synchronization method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data synchronization method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data synchronization method in any other appropriate manner (e.g., by means of firmware).

[0152] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0154] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0157] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0158] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0159] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data synchronization method, characterized in that: The method comprises: Obtain the rule engine and business processing rules based on the structured data of the migration source database; Generate a data processing task based on the rule engine, the business processing rule and the structured data; Dispatching the data processing task to a task processing server to process the structured data; The change data of the structured data is acquired by the change service capture server, and the structured data is synchronized in the migration target database based on the change data.

2. The method according to claim 1, characterized in that: The step of obtaining a rule engine and business processing rules based on structured data of the migration source database includes: Determine the migration source database in a preset database cluster, and select a structured data table in the migration source database as the structured data; Obtain rule engine selection information and business rule writing information input by the user; The preset rule engine corresponding to the rule engine selection information is used as the rule engine, and the business rule writing information is used as the business processing rule.

3. The method according to claim 1, characterized in that: The step of dispatching the data processing task to a task processing server to process the structured data includes: determining server loads of different task processing servers; The data processing tasks are allocated to the task queues of the task processing servers according to the server load, so that the task processing servers process the data processing tasks according to the order of the task queues.

4. The method according to claim 3, characterized in that: The step of causing the task processing server to process the data processing task according to the order of the task queue includes: Performing rule verification on the business processing rules of the data processing task by the task processing server; After the verification is passed, controlling the task processing server to load the rule engine of the data processing task; The structured data of the data processing task is processed according to the business processing rules through the rule engine of the task processing server.

5. The method according to claim 1, characterized in that: The acquiring the change data of the structured data by the change service capture server and synchronizing the structured data in the migration target database based on the change data includes: Using the change service capture server to read the transaction log of the migration source database, and extracting change data corresponding to the structured data from the transaction log, wherein the change data includes operation information and structured data table information of the structured data; The change service capture server performs data conversion on the structured data according to the data format of the migration target database; The structured data and the changed data are transmitted to the migration target database, so that the migration target database performs synchronization processing on the structured data according to the changed data.

6. The method according to claim 5, characterized in that: The transmitting the structured data and the changed data to the migration target database so that the migration target database performs synchronization processing on the structured data according to the changed data includes: Upload the structured data and the changed data as subscription topics to a Kafka message queue cluster; Distributing the structured data and the changed data to the migration target database subscribed to the subscription topic through the Kafka cluster message queue cluster; The structured data is synchronously processed using the migration target database according to the changed data.

7. The method according to claim 5, characterized in that: The transferring of the structured data and the changed data to the migration target database further includes: The structured data and the changed data are encrypted.

8. A data synchronization device, characterized in that: The device comprises: The processing preparation module is used to obtain the rule engine and business processing rules based on the structured data of the migration source database; A task construction module, used to generate a data processing task based on the rule engine, the business processing rules and the structured data; A task scheduling module, used for scheduling the data processing task to a task processing server to process the structured data; The change synchronization module is used to obtain the change data of the structured data through the change service capture server, and synchronize the structured data in the migration target database based on the change data.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data synchronization method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data synchronization method according to any one of claims 1 to 7 when executed.

Images