A log data synchronization task process control method and device

Abstract

The present invention discloses a method and device for controlling a log data synchronization task process, the device comprising: a console main application and an executor; the functions of the console main application comprising: executor registration, data source configuration management, data synchronization task management, and message queue topic management; the functions of the executor comprising: starting a log data synchronization process based on a registered executor, and reading change data to capture incremental log data under a topic partition of a message queue; setting subscription topics based on data source configuration management and data synchronization task management, and collecting and writing change data to capture incremental log data; and performing subscription and consumption concurrent operations of different topics in a message queue based on an executor group formed by multi-executor node deployment. The present invention can realize controllable, manageable, configurable, visualized, and clustered log data synchronization task process control, improve data collection efficiency, save manpower, and reduce production costs.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical big data, and in particular relates to a log data synchronization task process control method and device. Background Art

[0002] With the rapid development of medical informatization, medical big data has become an important driving force for promoting change and innovation in the medical industry. Especially when realizing cross-system and cross-platform data flow, log-based change data capture (CDC) technology has become a key means for efficient integration of medical big data because it can capture data changes in data sources in real time or near real time.

[0003] In the current market, tools such as Canal, Maxwell, Debezium, Flink CDC, Sqoop and Datax, as mainstream CDC solutions, generally face limitations such as complex configuration, high learning cost, high performance requirements and the need for authorized access to database logs. In addition, most CDC tools are deployed on a single machine and are difficult to expand to a cluster with multiple copies, increasing the risk of single point failure and task interruption. In the actual application of medical big data, there are many technical difficulties in the process of synchronously writing log data generated by the data source to the target source, which are specifically reflected in the following aspects:

[0004] (1) Inconvenient control and management of intermediate components: Administrators need to manually enter script commands in the background to view and control the synchronization status, which makes it difficult to batch manage log data synchronization tasks for multiple source tables. There is also a lack of intuitive and easy-to-use interface support for management operations such as starting and stopping tasks and selecting tables.

[0005] (2) The synchronization process lacks transparency and real-time performance: The log data synchronization process is a black box for implementers. In an emergency, if you need to quickly start or stop tasks or query task progress and status, you need to rely on the background operations of big data R&D personnel, which increases the complexity of troubleshooting and emergency response.

[0006] (3) Difficulty in monitoring the health of synchronization tasks: Effective monitoring of the status of CDC tasks (such as in progress, interrupted, and terminated) is crucial to ensuring the reliability and timeliness of data synchronization. Existing tools are insufficient in providing task health monitoring and it is difficult to reflect the health status of tasks in real time.

[0007] (4) Performance bottleneck of large table log data synchronization: In the medical system, some business tables may generate a large amount of log data, which may lead to task process interruption, inefficiency of the entire collection process, or even task interruption or termination during the log data synchronization process, thus interfering with the normal operation of the business system.

[0008] For data solution companies that have been working in the field of medical big data for many years, data collection scenarios are more diverse and complex, and there are some specific requirements:

[0009] (1) Data collection of medical systems deployed in multiple regions: For the logs generated by the end-to-end database table from the downstream institution database to the enterprise's internal clustered database, medical systems deployed in multiple regions (such as HIS, PASS, and inpatient systems, etc.) need to collect and report application data. This requires that the CDC solution must be able to support cross-regional and cross-system data integration.

[0010] (2) Diversity of heterogeneous data sources: The internal information system of the hospital as the data source often uses different database storage media (such as Oracle), while the internal digital center of the enterprise as the target source may use a more advanced distributed database (such as TiDB) for data storage. Therefore, the CDC solution needs to have the ability to collect and convert data across heterogeneous data sources.

[0011] (3) Eliminate data silos and realize the commercial value of data: Medical data is scattered among medical institutions at all levels, making it difficult to form a unified data view, which limits the commercial application potential of the data. It is necessary to use CDC technology to achieve real-time synchronization and integration of data, break down data silos, and provide support for applications such as disease population classification, large-screen display of population data, and disease management.

[0012] Therefore, in order to overcome the many challenges faced in data integration and meet industry-specific needs, it is necessary to develop more intelligent, efficient and reliable log-based CDC solutions. Summary of the invention

[0013] In view of the above, the purpose of the present invention is to provide a log data synchronization task process control method and device, to achieve controllable, manageable, configurable, visualized and clustered log data synchronization task process control, improve data collection efficiency, save manpower and reduce production costs, realize the start and stop of tasks, view the log data synchronization task status and visualize the health status through the interface of the console main application, improve the concurrency capability of the executor group by means of executor expandable nodes to cope with large-table log data synchronization tasks, suitable for log synchronization data integration deployed in multiple regions and real-time synchronization and integration of data in data collection scenarios across heterogeneous data sources, and contribute to the comprehensive management and application of medical big data.

[0014] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0015] An embodiment of the present invention provides a log data synchronization task process control device, comprising: a console main application and an executor;

[0016] The functions of the console main application include: executor registration, data source configuration management, data synchronization task management, and message queue topic management;

[0017] The functions of the executor include: starting the log data synchronization process based on the registered executor, and reading the change data to capture the incremental log data to the topic partition of the message queue; setting subscription topics based on data source configuration management and data synchronization task management, and collecting and writing the change data to capture the incremental log data; and forming an executor group based on multi-executor node deployment to perform concurrent subscription and consumption operations on different topics in the message queue.

[0018] Preferably, the data source configuration management function of the console main application includes:

[0019] In the data source configuration interface, you can enter the data source name, user name and password for authentication, select the data source driver, and data source connection information. The data source connection information includes the IP or port of the database corresponding to the read source and write source.

[0020] Preferably, the data synchronization task management function of the console main application includes:

[0021] In the task creation interface, you can click the New Task function button to create a new task and perform basic and advanced configurations. The basic configuration includes entering or selecting the task name, executor group name, synchronization type, read source connection information, and write source connection information. The synchronization type includes standby mode, precise mode, or mirror mode. The read source connection information includes the source instance of the read source and the read source database. The write source connection information includes the target instance of the write source and the write source database. The advanced configuration includes whether to automatically create a table, whether to enable snapshots, and / or whether to enable logical deletion.

[0022] Preferably, the data synchronization task management function of the console main application includes:

[0023] In the task control interface, you can view the task ID, task name, running status, executor group, incremental mode, number of exceptions and whether to take a snapshot. You can also start, pause, delete, edit, view details, reset the location or switch to incremental mode for each task. You can also click the task name function key to query the task, click the search function key to search the task, and click the new task function key to create a new task.

[0024] Preferably, the view details function includes:

[0025] In the task details interface, you can view the subtask ID, read the source database name, read the source table name, write the source database name, write the source table name, snapshot running status, incremental running status, and change data capture incremental status. It also supports performing sub-operations, deletions, and setting primary keys for each subtask. The sub-operations include controlling the start, stop, incremental shutdown, incremental compensation, incremental snapshot, abnormal threshold, and abnormal data volume of the selected subtask. It also supports entering the table name for table query, clicking the All function key to display all subtasks, clicking the To-be-compensated Tasks function key to view the tasks to be compensated, clicking the Change Data Capture Management function key to manage and set, clicking the Snapshot Management function key to set snapshots, clicking the Compensation Operation function key to set compensation, and clicking the Batch Start function key to batch control subtasks.

[0026] Preferably, the message queue topic management function of the console main application includes:

[0027] Create topic partitions for message queues according to different log types or different data sources, initialize the task process to create topic information, initialize the subscription process to listen to topic information, and write the corresponding message queue topic name and task status including start, pause or delete into the distributed cache middleware according to the start, pause or delete operations of the task executed in the task control interface.

[0028] Preferably, the executor based on the completed registration starts the log data synchronization process, and reads the change data to capture the incremental log data under the topic partition of the log message queue, including:

[0029] After completing the executor registration in the executor registration service center of the console main application, query the task status corresponding to different topics of the message queue according to the API provided by the distributed cache middleware. When the task status is started, the task process and the subscription process enter the running state from the ready state, and the executor node starts the log data synchronization task process to start real-time reading of change data, capture incremental log data, and write it to the topic partition of the message queue. When the task status is paused, the task process and the subscription process enter the blocking state and give up the CPU execution right. When the task status is deleted, the task process and the subscription process enter the terminated state and delete the topic information and the data under the topic partition.

[0030] Preferably, the setting of subscription topics based on data source configuration management and data synchronization task management, and the collection and writing of change data capture incremental log data; and the formation of executor groups based on multi-executor node deployment, performing concurrent operations of subscription and consumption of different topics in the message queue, include:

[0031] After the executor node starts the log data synchronization task process, according to the specific settings in the data source configuration management and data synchronization task management, the executor node subscribes to the change data under the topic partition in the message queue that needs to be collected to capture the incremental log data, connects the executor node to the write source database through JDBC and executes the corresponding SQL statement to write the collected real-time data into the corresponding write source database;

[0032] Multiple executor nodes are combined into an executor group. Based on the executor group, multiple subscription processes are used to concurrently collect the subscription consumption of the change data capture incremental log data under the topic partition in the message queue, and the subsequent operation of writing the collected data to the write source database in real time is executed accordingly.

[0033] Preferably, the relevant APIs encapsulated in the Java program based on the Redis distributed cache middleware implement actuator node registration, actuator node heartbeat maintenance and expired actuator node cleaning.

[0034] To achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a log data synchronization task process control method, which is implemented by using the above-mentioned log data synchronization task process control device, and includes the following steps:

[0035] Use the console main application to register executors, manage data source configuration, manage data synchronization tasks, and manage message queue topics;

[0036] The executor starts the log data synchronization process based on the registered executor, and reads the change data to capture the incremental log data to the topic partition of the message queue; sets the subscription topic based on the data source configuration management and data synchronization task management, and collects and writes the change data to capture the incremental log data; and forms an executor group based on the deployment of multiple executor nodes to perform concurrent subscription and consumption operations on different topics in the message queue.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] (1) The present invention can be viewed and operated through the visual interface of the console main application, which can realize controllable, manageable, configurable and traceable log data synchronization task status and clustered log data synchronization task process control, which can greatly improve the work efficiency of data collection personnel and reduce production costs.

[0039] (2) The present invention supports real-time deployment of multiple executor nodes, so that when the sudden log data accumulation is too large and the consumption capacity is insufficient during the log data synchronization process, the data consumption capacity can be improved, the data collection efficiency can be improved, and the data synchronization time can be reduced through the flexible deployment of the executor group.

[0040] (3) The present invention supports dynamic expansion and contraction of actuator nodes, and expansion of nodes, which can solve the problem of single node failure and improve the system throughput (data volume / time consumption). It supports node contraction to reduce the number of actuator nodes, thereby saving server (CPU, memory, disk, etc.) resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 It is a schematic diagram of the structure and flow of a log data synchronization task process control device provided by an embodiment of the present invention;

[0043] Figure 2 is a schematic diagram of a task creation interface provided by an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of a task control interface provided by an embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of a task details interface provided by an embodiment of the present invention;

[0046] Figure 5 It is a process status diagram of a task process and a subscription process provided by an embodiment of the present invention;

[0047] Figure 6 It is a schematic diagram of implementing actuator node registration, actuator node heartbeat maintenance and expired actuator node cleanup based on Redis provided by an embodiment of the present invention;

[0048] Figure 7 It is a flow chart of a log data synchronization task process control method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0050] The inventive concept of the present invention is: in view of the problems of inconvenient control and management of intermediate components of CDC log data synchronization tasks in the prior art, lack of transparency and real-time performance of the synchronization process, difficulty in monitoring the health of synchronization tasks, and performance bottlenecks of large-table log data synchronization, the embodiments of the present invention provide a log data synchronization task process control method and device, which realizes the start and stop of tasks, the viewability of log data synchronization task status and the visual health status through the interface of the console main application, and improves the concurrency capability of the executor group by means of executor scalable nodes to cope with large-table log data synchronization tasks, thereby realizing controllable, manageable, configurable, visualized and clustered log data synchronization task process control, improving data collection efficiency, saving manpower and reducing production costs, and being suitable for log synchronization data integration deployed in multiple regions and real-time synchronization and integration of data in data collection scenarios across heterogeneous data sources, which is conducive to the comprehensive management and application of medical big data.

[0051] Figure 1 1 is a schematic diagram of the structure and flow of a log data synchronization task process control device provided by an embodiment of the present invention. Figure 1 As shown, in the embodiment, based on the Java Spring framework system, the front end relies on the Vue front-end technology framework, and a log data synchronization task process control device (named Tiger) based on the change data capture (CDC) log data synchronization collection is built. The core includes the console main application (named Tiger-admin-console) and the executor (named Tiger-task-executor). The two roles perform their respective duties and play different functions.

[0052] In the embodiment, the functions of the console main application include: executor registration, data source configuration management, data synchronization task management, and message queue (kafka) topic management (topic is similar to a table in a database).

[0053] Specifically, the executor registration function includes: completing executor registration through an executor registration center.

[0054] Specifically, the data source configuration management function includes: in the data source configuration interface, it supports inputting the data source name, user name and password for authentication, selecting the data source driver, and data source connection information, wherein the data source connection information includes the IP or port of the database corresponding to the read source and write source.

[0055] Specifically, the data synchronization task management functions include: Figure 2As shown, in the task creation interface (state = create), you can click the new task function button to create a new task and perform basic and advanced configurations. The basic configuration includes entering or selecting the task name, executor group name, synchronization type, read source connection information, and write source connection information. The synchronization type includes standby mode, precise mode, or mirror mode. The read source connection information includes the source instance of the read source and the read source database. The write source connection information includes the target instance of the write source and the write source database. The advanced configuration includes whether to automatically create a table, whether to enable snapshots, and / or whether to enable logical deletion. Add test connections after the source instance and target instance to test and verify whether the connection information, account number, password, and other information of the source instance, target instance, and database of the prerequisite are connected.

[0056] The data synchronization task management function also includes: Figure 3 As shown, in the task control interface, you can view the task ID (named TASK_01, TASK_02, ...), task name (format: read source database-write source database, where the read source database includes ORACLE, CACHE, MYSQL and SQLSERVER, etc., and the write source database includes TIDB and MONGODB, etc.), running status (including start, run, pause), executor group name (format: write source database_executor, such as TIDB_EXCUTOR1, TIDB_EXCUTOR2, ...), incremental mode (including CDC mode and compensation mode), number of exceptions and whether to snapshot. It also supports the execution of operations including start, pause, delete, edit, view details, reset site (high site switch to low site action) or switch incremental mode for each task. It also supports clicking the task name function key to query the task, clicking the search function key to search the task, and clicking the new task function key to create a new task.

[0057] The data synchronization task management function also includes: Figure 4As shown, in the task details interface, you can view the subtask ID, read the source database name (such as SQLUser), read the source table name (such as bmsql_order_line, bmsql_config, bmsql_history, bmsql_customer, bmsql_warehouse, bmsql_stock, etc.), write the source database name (such as core_fact), write the source table name (the uppercase form corresponding to the read source table name), snapshot running status, incremental running status, CDC incremental status, and support operations including sub-operations, deletion, and setting primary keys for each subtask. The sub-operations include controlling the start, stop, incremental shutdown, incremental compensation, incremental snapshot, abnormal threshold, and abnormal data volume of the selected subtask. It also supports entering the table name for table query, clicking the All function key to display all subtasks, clicking the To-be-executed Compensation Task function key to view the To-be-compensated tasks, clicking the CDC Management function key to perform management settings, clicking the Snapshot Management function key to perform snapshot settings, clicking the Compensation Operation function key to perform compensation settings, and clicking the Batch Start function key to perform batch control of subtasks.

[0058] Specifically, the message queue topic management function includes: creating topic partitions of the message queue according to different log types or different data sources, initializing the task process for creating topic information, initializing the subscription process for listening to topic information, and writing the corresponding message queue topic name and the task status including start, pause or delete into the distributed cache middleware Redis according to the start, pause or delete operations of the task executed in the task control interface.

[0059] Redis is a type of non-relational database (NoSQL). It is an open source, memory-based, high-performance key-value storage system that supports multiple types of data structures. The use of distributed cache components (such as Redis) can support sharing of memory object data among multiple executor nodes, which is impossible with JVM (Java Virtual Machine)-based cache data structures (such as Map) because JVM's memory is isolated. Distributed cache components not only support cross-node sharing, but also provide a rich variety of data types and collection operations. The set and get methods in the API (Application Interface) provided by the distributed cache middleware Redis can be used to maintain, manage and query the status of different topics stored in hash data structures. These hash data structures can be regarded as a collection similar to Map in Redis, allowing the values ​​associated with different message topics to be quickly set and retrieved by key. The executor program manages its pause (stop), start (start), delete (delete), incremental compensation (incre-start) and incremental shutdown (incre-stop) through the state (state) field. The executor program queries the status of each topic in the shared cache collection Map to determine the execution of different operations.

[0060] In the embodiment, the functions of the executor include: starting the log data synchronization process based on the executor that has completed registration, and reading the CDC incremental log data to the topic partition of the message queue; setting the subscription topic based on the data source configuration management and data synchronization task management, and collecting and writing the CDC incremental log data; and forming an executor group (group) based on the deployment of multiple executor nodes to perform concurrent operations of subscribing and consuming different topics in the message queue. The executor collects and reads the source CDC incremental log data and writes it to the corresponding topic partition directory of the message queue created in the form of table name (formatted) naming (that is, storing the read source log data to the corresponding disk file directory), and constructs the format of the topic name = the executor group name of the task + the task ID + the read source table name of the task + the uppercase read source database name.

[0061] Specifically, after completing the executor registration in the executor registration service center of the console main application, query the task status corresponding to different topics of the message queue according to the API provided by the distributed cache middleware Redis. Figure 5As shown in the figure, when the task state is started (state = start), the task process and the subscription process enter the running state from the ready state, and the executor node starts the log data synchronization task process to start real-time reading of CDC incremental log data (different reading source databases include Oracle, MySQL, and MongoDB, etc.) and write it to the topic partition of the message queue (partitions are essentially files). When the task state is paused (state = stop), the task process and the subscription process enter the blocking state and give up the CPU execution right. The task process no longer reads the CDC incremental log data of the data source, and the subscription process also enters the blocking state and no longer consumes the data under the topic partition of the message queue. It will regain the CPU execution right after waiting for an event to occur (the variable is set to state = start). When the task state is deleted (state = deleted), the task process and the subscription process enter the terminated state, the program no longer reads the CDC incremental log data of the source, the subscription process no longer consumes the data under the topic partition of the message queue, and the corresponding topic is also deleted.

[0062] When the task state is started (state=start), after the executor node starts the log data synchronization task process, according to the specific settings in the data source configuration management and data synchronization task management, the executor node subscribes to and consumes the CDC incremental log data under the topic partition in the message queue that needs to be collected, connects the executor node to the write source database through JDBC (Java Database Connectivity, an API for Java programs to access relational databases) and executes the corresponding SQL (Structured Query Language, a database query and programming language) statements (such as insert into, delete from, update xx (table), etc.), and writes the collected real-time data to the corresponding write source database (such as TiDB, MySQL, MongoDB, etc.). Furthermore, by combining multiple executor nodes into multi-copy nodes and building an executor group, the subscription consumption and concurrent collection of CDC incremental log data under the topic partition in the message queue are performed based on the executor group through multiple subscription processes, and the subsequent operation of writing the collected data to the write source database in real time is executed accordingly.

[0063] In the embodiment, when a log data synchronization task has a high delay, the following sudden abnormal situations occur: the subscription process has limited concurrency in consuming data under the topic partition, resulting in slow consumption, or a certain table has a sharp increase in incremental log data, resulting in the task process writing data at a much faster speed than the subscription process consumption speed. The preferred solution is:

[0064] (1) By deploying multiple executor nodes to combine multiple replica nodes to form a consumer group, multiple subscription processes can improve the corresponding concurrent consumption of message queue topics. When multiple executor nodes consume messages under a topic partition, according to the principle that there is only one consumer for a partition and a consumer can consume multiple partitions, the consumption capacity is greatly improved from the previous one executor node as a consumer consuming multiple partitions to multiple consumers and each consumer consumes log data under different partitions of a topic.

[0065] (2) If Figure 6 As shown, based on the Redis system, actuator node registration, actuator node heartbeat maintenance and expired actuator node cleaning are implemented to improve the high availability, high scalability and high load capacity of the actuator nodes.

[0066] Executor node registration: Use Redis to provide a hash set setting (hset) command, and combine to construct a node unique key (application name + node IP + port), which is used to send a registration (register) event to the registration service center of the console main application to notify the executor node of its survival (online).

[0067] Executor node heartbeat maintenance: A mechanism is adopted to periodically update the expiration time of all registered executor nodes to ensure the heartbeat of the executor nodes is kept alive.

[0068] Expired executor node cleanup: Pull the node list in real time, traverse and check the expiration time of all registered executor nodes, if the expiration time is less than the current time, the executor node is considered expired and set to offline. The offline operation can remove the node information registered on the Redis component and the node information of the registration service center, and manage the tasks being executed and the suspension operation of tasks to be executed of the executor node.

[0069] In summary, a log data synchronization task process control device provided by an embodiment of the present invention can realize controllable, manageable, configurable, visualized and clustered CDC log data synchronization through the interface design of the console main application and the linkage execution of the log data synchronization task, further improve the data collection efficiency, save manpower and reduce production costs, and is suitable for log synchronization data integration deployed in multiple regions and real-time synchronization and integration of data in data collection scenarios across heterogeneous data sources, which is conducive to the comprehensive management and application of medical big data.

[0070] Based on the same inventive concept, Figure 7 As shown, the embodiment of the present invention also provides a log data synchronization task process control method, comprising the following steps:

[0071] S1, register executors, manage data source configuration, manage data synchronization tasks, and manage message queue topics through the console main application.

[0072] S2, starts the log data synchronization process based on the registered executor through the executor, and reads the change data to capture the incremental log data to the topic partition of the message queue; sets the subscription topic based on the data source configuration management and data synchronization task management, and collects and writes the change data to capture the incremental log data; and forms an executor group based on the deployment of multiple executor nodes to perform concurrent subscription and consumption operations on different topics in the message queue.

[0073] It should be noted that the log data synchronization task process control method provided in the above embodiment and the log data synchronization task process control device belong to the same inventive concept, and its specific implementation process is detailed in the log data synchronization task process control device embodiment, which will not be repeated here.

[0074] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A log data synchronization task process control device, characterized in that: include: Console main application and executor; The functions of the console main application include: executor registration, data source configuration management, data synchronization task management and message queue topic management; in the data synchronization task management function of the console main application, it includes: in the task control interface, it supports viewing the task ID, task name, running status, executor group, incremental mode, number of exceptions and whether it is a snapshot, and also supports the execution of each task including starting, pausing, deleting, editing, viewing details, resetting the position or switching the incremental mode. It also supports clicking the task name function key to query the task, clicking the search function key to search the task, and clicking the new task function key to create a new task; Among them, in the view details function, it includes: in the task details interface, it supports viewing the subtask ID, reading the source data Database name, read source table name, write source database name, write source table name, snapshot running status, incremental running status, change data capture incremental status. It also supports sub-operations, deletion and primary key setting for each sub-task. Sub-operations include controlling the start, stop, incremental shutdown, incremental compensation, incremental snapshot, abnormal threshold and abnormal data volume of the selected sub-task. It also supports entering table name for table query, clicking the All function key to display all sub-tasks, clicking the To-be-executed Compensation Task function key to view the To-be-compensated tasks, clicking the Change Data Capture Management function key for management settings, clicking the Snapshot Management function key for snapshot settings, clicking the Compensation Operation function key for compensation settings, and clicking the Batch Start function key for batch control of sub-tasks. The functions of the executor include: starting the log data synchronization process based on the executor that has completed registration, and reading the change data to capture the incremental log data to the topic partition of the message queue; setting the subscription topic based on the data source configuration management and data synchronization task management, and collecting and writing the change data to capture the incremental log data; and forming an executor group based on the deployment of multiple executor nodes, performing concurrent operations of subscription and consumption of different topics in the message queue. When multiple executor nodes consume messages under the topic partition, according to the principle that one partition has only one consumer and one consumer consumes multiple partitions at the same time, the previous executor node is used as a consumer to consume multiple partitions, and then expanded to multiple consumers and each consumer consumes log data under different partitions of a certain topic.

2. The log data synchronization task process control device according to claim 1, characterized in that: The data source configuration management function of the console main application includes: In the data source configuration interface, you can enter the data source name, user name and password for authentication, select the data source driver, and data source connection information. The data source connection information includes the IP or port of the database corresponding to the read source and write source.

3. The log data synchronization task process control device according to claim 1, characterized in that: The data synchronization task management function of the console main application includes: In the task creation interface, you can click the New Task function button to create a new task and perform basic and advanced configurations. The basic configuration includes entering or selecting the task name, executor group name, synchronization type, read source connection information, and write source connection information. The synchronization type includes standby mode, precise mode, or mirror mode. The read source connection information includes the source instance of the read source and the read source database. The write source connection information includes the target instance of the write source and the write source database. The advanced configuration includes whether to automatically create a table, whether to enable snapshots, and / or whether to enable logical deletion.

4. The log data synchronization task process control device according to claim 1, characterized in that: The message queue topic management function of the console main application includes: Create topic partitions for message queues according to different log types or different data sources, initialize the task process to create topic information, initialize the subscription process to listen to topic information, and write the corresponding message queue topic name and task status including start, pause or delete into the distributed cache middleware according to the start, pause or delete operations of the task executed in the task control interface.

5. The log data synchronization task process control device according to claim 4, characterized in that: The executor based on the completed registration starts the log data synchronization process and reads the change data to capture the incremental log data under the topic partition of the message queue, including: After completing the executor registration in the executor registration service center of the console main application, query the task status corresponding to different topics of the message queue according to the API provided by the distributed cache middleware. When the task status is started, the task process and the subscription process enter the running state from the ready state, and the executor node starts the log data synchronization task process to start real-time reading of change data, capture incremental log data, and write it to the topic partition of the message queue. When the task status is paused, the task process and the subscription process enter the blocking state and give up the CPU execution right. When the task status is deleted, the task process and the subscription process enter the terminated state and delete the topic information and the data under the topic partition.

6. The log data synchronization task process control device according to claim 5, characterized in that: The setting of subscription topics based on data source configuration management and data synchronization task management, and collection and writing of change data capture incremental log data; And based on the deployment of multiple executor nodes to form an executor group, concurrent subscription and consumption operations of different topics in the message queue are performed, including: After the executor node starts the log data synchronization task process, according to the specific settings in the data source configuration management and data synchronization task management, the executor node subscribes to the change data under the topic partition in the message queue that needs to be collected to capture the incremental log data, connects the executor node to the write source database through JDBC and executes the corresponding SQL statement to write the collected real-time data into the corresponding write source database; Multiple executor nodes are combined into an executor group. Based on the executor group, multiple subscription processes are used to concurrently collect the subscription consumption of the change data capture incremental log data under the topic partition in the message queue, and the subsequent operation of writing the collected data to the write source database in real time is executed accordingly.

7. The log data synchronization task process control device according to claim 1, characterized in that: Based on the relevant API encapsulated in the Java program by the Redis distributed cache middleware, actuator node registration, actuator node heartbeat maintenance and expired actuator node cleaning are implemented.

8. A log data synchronization task process control method, implemented by using the log data synchronization task process control device according to any one of claims 1 to 7, characterized in that: The following steps are involved: Use the console main application to register executors, manage data source configuration, manage data synchronization tasks, and manage message queue topics; The executor starts the log data synchronization process based on the registered executor, and reads the change data to capture the incremental log data to the topic partition of the message queue; Set up subscription topics based on data source configuration management and data synchronization task management, and collect and write change data capture incremental log data; And based on the deployment of multiple executor nodes to form an executor group, concurrent operations of subscription and consumption of different topics in the message queue are performed.

Images