Data synchronization management system and method based on configuration abstraction and policy automatic construction
By using a unified task description model and an automatic strategy building layer, the problems of complex configuration and fragmented task management in the Kafka Connect data synchronization solution are solved, achieving business-friendly data synchronization management, reducing the configuration threshold and improving task creation efficiency and system maintainability.
Patent Information
- Application Number
- CN202611267709.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-20
- Publication Date
- 2026-11-10
AI Technical Summary
Existing Kafka Connect data synchronization solutions are complex to configure, have a high barrier to entry, are designed for developers, involve a large number of underlying technical parameters, and make it difficult for business users to configure them; the source and target connectors are configured separately, increasing the probability of errors; standard synchronization tasks and extended processing tasks are managed separately, lacking a unified task model and lifecycle control.
This paper provides a data synchronization management system based on configuration abstraction and automatic strategy construction. It incorporates source, target and extended tasks into the same task object through a unified task description model, and uses the automatic strategy construction layer to automatically map business-level input parameters to underlying execution parameters, so as to realize unified task creation, start, pause, deletion and status management.
It significantly reduces the barrier to entry and learning cost of creating data synchronization tasks, reduces configuration errors, achieves unified management of standard synchronization tasks and extended tasks, and improves the maintainability and scalability of the system.
Smart Images

Figure CN123068030A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data integration and task orchestration technology, specifically to a data synchronization management system and method based on configuration abstraction and automatic strategy construction, which is particularly suitable for unified task management and automatic configuration generation for business personnel in data synchronization scenarios based on the Kafka Connect framework. Background Technology
[0002] Apache Kafka Connect is a widely used data integration framework for scalable and reliable data migration between Apache Kafka and other systems. Kafka Connect coordinates data flows through a high-level abstraction called a Connector, where each Connector instance manages data replication tasks between Kafka and another system. Users can configure and manage Connector instances via the Kafka Connect REST API or property files.
[0003] In practical applications, a complete data synchronization chain typically requires configuring a source connector and a sink connector separately. The source connector is responsible for listening to or pulling changed data from the source database and writing it to a Kafka topic, while the sink connector is responsible for reading data from the Kafka topic and writing it to the target database. Taking the JDBC connector as an example, its configuration items include dozens of parameters such as connection address, authentication information, table filtering rules, polling interval, batch size, incremental column name, and snapshot mode, and the configuration parameters differ significantly for different database types (MySQL, PostgreSQL, SQL Server, etc.). When other data sources such as MongoDB and Elasticsearch are involved, the configuration parameter system is also very different.
[0004] With the widespread adoption of Kafka Connect in production environments, its configuration complexity has become a major obstacle affecting efficiency. Analysis has revealed at least the following issues with existing technologies: First, the configuration threshold is high and the target audience is limited. Kafka Connect's configuration parameters are designed for developers or operations personnel, involving a large number of low-level technical parameters such as connector class names, transformers, number of tasks, offset storage topics, and state storage topics. For implementers who only need to express the business intent of "listening to which table in which database and synchronizing to which table in which database," they must deal with a large number of technical details unrelated to business semantics, resulting in a high threshold for task creation, a high learning cost, and a high error rate.
[0005] Second, the Source and Sink tasks are configured separately. In the existing model, the source connector and the target connector need to be configured and managed independently. Users must separately understand the source connection parameters, target write parameters, and topic binding relationships, and manually ensure that the topic names, data formats, etc., are consistent between the two. This fragmented configuration method increases the complexity of configuration and the probability of errors.
[0006] Third, standard synchronization tasks and extended processing tasks are managed separately. In actual implementation, enterprises often need to add additional processing logic outside the standard data synchronization chain. For example, they may need to assemble data in a topic according to a template and submit it to an HTTP interface, append business tags, inject extended fields, or perform custom processing. This kind of extended logic is usually implemented in a separate program, lacking a unified task model and control entry point with the standard synchronization tasks, making it difficult to achieve complete lifecycle management such as creation, pausing, deletion, editing, and status write-back.
[0007] Fourth, Kafka Connect's lifecycle management capabilities for tasks are limited. Although Kafka Connect provides a REST API for managing connectors, users still need to perform operations such as task creation, startup, pause, and deletion through underlying HTTP interface calls or property file operations. For running tasks, it is difficult to update configurations and manage their state through a unified business-level control interface.
[0008] In summary, existing data synchronization solutions have significant shortcomings in terms of configuration abstraction, unified task description, extended task management, and lifecycle control. There is an urgent need for a system and method that can lower the barrier to entry, enable automatic configuration and construction, and provide unified management of standard and extended tasks. Summary of the Invention To address the problems existing in the aforementioned background technology, this invention provides a data synchronization management system and method based on configuration abstraction and automatic strategy construction.
[0009] This invention provides a data synchronization management system based on configuration abstraction and automatic policy construction, comprising: The business configuration end is used to receive task description information input by the user; The management application service layer communicates with the business configuration layer and is used to receive the task description information and construct a unified task object. The automatic strategy construction layer communicates with the management application service layer and is used to determine the corresponding strategy template according to the role and data source type of each connector in the unified task object, and map the business-level input parameters in the task description information to the strategy template to automatically generate the underlying execution parameters. The Kafka Connect execution layer communicates with the policy auto-construction layer to receive the underlying execution parameters and execute standard data synchronization tasks. Hook the execution layer to perform extended processing tasks; The task orchestration and lifecycle management module communicates with the management application service layer, the KafkaConnect execution layer, and the Hook execution layer, respectively, and is used to perform unified creation, start, pause, deletion, status query, and configuration editing operations on standard synchronous tasks and extended tasks. The task definition storage module is communicatively connected to the management terminal application service layer and is used to store the unified task object.
[0010] As a preferred embodiment of this application, the unified task object includes at least a task name, a connector set (connectors), and an extended task set (hookConfigs); wherein, the connector set (connectors) is used to describe standard synchronization tasks, and each connector object includes at least a name, a role, a data source type (dataSource), a state, a common parameter set (common), and an enhanced parameter set (options); the role is used to identify whether the connector is a source role or a target role.
[0011] As a preferred embodiment of this application, the common parameter set is used to accommodate standard connection parameters, including but not limited to host address, port, username, password, database name, table name, topic prefix, and task concurrency; the enhanced parameter set options is used to accommodate connector-specific or enhanced configurations, including but not limited to snapshot mode, deletion policy, replacement policy, and mode change switch.
[0012] As a preferred embodiment of this application, the enhanced parameter set options is further provided with a fields substructure, which serves as a carrier area for front-end custom attribute information, used to carry business tags, extended metadata or route identifiers; the custom attributes in fields can be read and used in subsequent template construction, hook processing, interface delivery, audit records or downstream extended logic.
[0013] As a preferred embodiment of this application, the automatic strategy construction layer includes: The strategy template storage unit is used to store strategy templates corresponding to different data source types and different roles; The strategy matching unit is used to match the corresponding strategy template based on the role of the connector and the data source type dataSource in the unified task object. The parameter mapping unit is used to map the business-level input parameters in the task description information to the strategy template, automatically complete the default values, generate the topic name, construct the source end listening range, generate the target end writing parameters, and predefine and optimize the deletion strategy, snapshot mode, replacement strategy and task concurrency. The template extension unit is used to extend the corresponding data source strategy template when a new database type is connected.
[0014] As a preferred embodiment of this application, the Hook execution layer includes: The configuration reading unit is used to read the topic list, consumer group identifier, task status, request body template, target HTTP address and request header parameters from the hookConfigs of the unified task object; The message subscription unit is used to subscribe to a specified Kafka topic and retrieve messages; The template rendering unit is used to map the keys, values, topics, and timestamps in the message to the request body template according to the template variable replacement rules, thereby constructing the HTTP request body; The request delivery unit is used to deliver the constructed HTTP request body to an external business interface or trigger custom extension logic.
[0015] As a preferred embodiment of this application, the task orchestration and lifecycle management module includes: The task creation unit is used to create or update the corresponding source task and sink task in the Kafka Connect execution layer according to the unified task object, and to create the corresponding hook task in the Hook execution layer. The task status management unit is used to obtain the running status of source tasks and sink tasks through the REST API of the Kafka Connect execution layer, and to obtain the running status of hook tasks, and synchronize the running status back to the unified control plane. The task control unit is used to start, pause, delete, and update the configuration of source, sink, and hook tasks.
[0016] This invention also provides a data synchronization management method based on configuration abstraction and automatic policy construction, the method being applied to the system described above, the method comprising the following steps: S1. Receive a task request. The task request is initiated by business personnel through the business configuration terminal. The task request contains task description information for data synchronization, including at least the source database type, source database name, source table name, target database type, target database name, and target table name. S2. The management application service layer constructs a unified task object based on the task description information, that is, it parses and generates a unified task description model. The unified task object includes at least a connector set (connectors) and an extended task set (hookConfigs). Each connector object includes at least a role, a data source type (dataSource), a common parameter set (common), and an enhanced parameter set (options). S3. The strategy automatic construction layer automatically constructs the underlying execution parameters based on the unified task object, generates corresponding source parameters and sink parameters for each connector in the unified task object, and generates the parameters required for hook execution for hookConfigs. S4. The task orchestration and lifecycle management module automatically builds the underlying execution parameters of the source generated by the strategy in step S3, creates or updates the corresponding source task in the Kafka Connect execution layer, and establishes a source-end listening or change capture link. S5. The task orchestration and lifecycle management module automatically builds the underlying execution parameters of the sink generated by the strategy in step S3, creates or updates the corresponding sink task in the Kafka Connect execution layer, binds the topic and the target table, and forms a complete synchronous link from the source database to the message topic and then to the target database. S6. The task orchestration and lifecycle management module creates a corresponding hook task in the Hook execution layer based on the hookConfigs in the unified task object, registers hook consumer instances, subscribes to specified topics, and constructs an HTTP request body according to the template to execute extended processing logic. S7, the task orchestration and lifecycle management module performs unified runtime control operations on source tasks, sink tasks, and hook tasks, including starting, pausing, deleting, and configuration editing; The S8 task orchestration and lifecycle management module performs unified monitoring and backtracking operations on source tasks, sink tasks, and hook tasks, records status, logs, and audit information, and implements retry strategies for tasks that fail to be delivered.
[0017] As a preferred embodiment of this application, step S3 specifically includes: S31. The automatic strategy construction layer extracts the role and data source type (dataSource) of each connector from the unified task object; S32. Match the corresponding strategy template from the strategy template storage unit according to the role and the data source type dataSource; S33. Map the business-level parameters in the common parameter set to the corresponding fields of the strategy template; S34. Automatically fill in the default value according to the data source type and the role, generate the topic name, and construct the source end listening range or the target end writing parameters; S35. Predefined optimizations are performed on deletion strategies, snapshot modes, replacement strategies, and task concurrency to generate complete underlying execution parameters.
[0018] As a preferred embodiment of this application, step S4 specifically includes: S41. The task orchestration and lifecycle management module submits the source connector configuration through the Kafka Connect REST API; S42, the Kafka Connect execution layer starts the source task based on this configuration, establishing a source-side listening or change capture link.
[0019] As a preferred embodiment of this application, step S5 specifically includes: S51, the task orchestration and lifecycle management module submits the sink connector configuration through the Kafka Connect REST API; The S52 and Kafka Connect execution layer start the sink task according to this configuration; S53. Bind the topic and the target table to form a complete synchronization link from the source database to the message topic and then to the target database.
[0020] As a preferred embodiment of this application, step S6 specifically includes: S61. Read the topic list, consumer group identifier, request body template, target HTTP address and request header parameters from the hookConfigs of the unified task object; S62. The Hook execution layer subscribes to the specified Kafka topic and retrieves messages; S63. Map the keys, values, topics, and timestamps in the message to the request body template according to the template variable substitution rules, and construct the HTTP request body according to the template; S64. Deliver the constructed HTTP request body to the external business interface or trigger custom extension logic.
[0021] As a preferred embodiment of this application, in step S7, the configuration editing operations for the source task and sink task include: resubmitting the updated strategy result to the Kafka Connect execution layer to complete the configuration update and status write-back; the configuration editing operations for the hook task include: editing the request template, target address, request header and custom attribute mapping and re-issuing them.
[0022] As a preferred embodiment of this application, step S8 specifically includes: S81. The task status management unit obtains the running status of the source task and sink task through the REST API of the Kafka Connect execution layer, and obtains the running status of the hook task through the internal interface, and synchronizes the running status back to the unified control plane. S82. The monitoring and auditing module records the operation logs and running logs of all tasks in the system; S83. The monitoring and auditing module implements a retry strategy for hook tasks that fail to be delivered. S84. All task status information, log information, and audit information are displayed uniformly on the management interface.
[0023] The present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the method described in any of the preceding claims.
[0024] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method described in any of the preceding claims.
[0025] The beneficial effects of this invention are: (1) This invention uses a unified task description model to describe the source connector, target connector and extended tasks in the same task object. Business personnel only need to provide the source library, source table, target library, target table and a few business-level attributes to complete the task definition. There is no need to deal with the large number of low-level configuration parameters of Kafka Connect for developers, which significantly reduces the threshold and learning cost of creating data synchronization tasks.
[0026] (2) This invention automatically constructs a strategy layer, automatically matches the corresponding strategy template according to the role (source / sink) of the connector and the data source type, and automatically maps the business-level input parameters to the underlying execution parameters, automatically fills in the default values, generates topic names, constructs the listening range and writes parameters, reduces manual configuration errors, and improves task creation efficiency and configuration consistency.
[0027] (3) By incorporating standard synchronous tasks (source / sink) and hook extended tasks into a unified task model and a unified lifecycle management interface, this invention enables unified creation, startup, pause, deletion, status query and configuration editing of the two types of tasks, solving the problem of decentralized management of standard synchronous tasks and extended processing tasks in the prior art, and facilitating centralized management and auditing.
[0028] (4) By setting the fields substructure under the enhanced parameter set options, this invention provides a carrying area for front-end custom attribute information. Without destroying the standard connection parameter structure, it retains the flexible extension injection capability for the business side, so that custom attributes such as business tags, extended metadata, and route identifiers can be read and used in template construction, hook processing, and interface delivery.
[0029] (5) This invention supports editing and re-issuing the configuration content of running tasks through a unified lifecycle control interface on the management end. For source and sink tasks, the updated strategy results can be resubmitted to the KafkaConnect execution layer to complete the configuration update. For hook tasks, it supports editing and updating the request template, target address, and other content, thereby improving the maintainability of the system.
[0030] (6) By adopting a templated extension method in the strategy construction layer, when a new database type needs to be connected, the corresponding data source strategy template can be extended in the strategy construction layer to complete the automatic construction using the unified task model. It has the ability to extend the data source within the scope supported by Kafka Connect.
[0031] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description
[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0033] Figure 1 This is a schematic diagram of the overall architecture of a data synchronization management system based on configuration abstraction and automatic policy construction according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a unified task description model according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating the task creation and execution logic of an embodiment of the present invention; Figure 4 This is a schematic diagram of the automatic strategy construction process according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the execution flow of a Hook extended task according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the lifecycle management process according to an embodiment of the present invention; Figure 7 This is a flowchart illustrating a data synchronization management method based on configuration abstraction and automatic policy construction, according to an embodiment of the present invention. Detailed Implementation
[0034] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0035] It should be noted that the descriptions involving "first," "second," etc., in this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention.
[0036] It should be noted that, in the embodiments of the present invention, Kafka Connect is a component of Apache Kafka used for scalable and reliable data integration between Kafka and other data systems. It implements data import and export through the abstraction of connectors and tasks.
[0037] The Source connector is a connector in Kafka Connect responsible for reading data from external data sources (such as databases) and writing it to Kafka topics.
[0038] Sink connector (target connector): It is a connector in Kafka Connect that is responsible for reading data from Kafka topics and writing it to external data targets (such as databases).
[0039] CDC (Change Data Capture): A technology used to identify and capture data that has changed in a data source.
[0040] Hook tasks refer to extended processing tasks that are attached outside the standard data synchronization link, used to perform additional processing on the data in the topic, such as HTTP interface delivery, message redelivery, etc.
[0041] The core idea of the data synchronization task management system and method provided by this invention is to incorporate the originally scattered source configuration, sink configuration and hook extended configuration into the same task object through a unified task description model, and to automatically convert business-level semantics into underlying technical parameters through an automatic strategy construction layer, thereby significantly reducing the configuration threshold and management complexity of data synchronization tasks.
[0042] The technical solution of the present invention will be described in detail below with reference to specific embodiments.
[0043] Example 1 Please see Figure 1 This invention provides a data synchronization management system based on configuration abstraction and automatic strategy construction.
[0044] Figure 1 This is a schematic diagram of the overall architecture of a data synchronization management system based on configuration abstraction and automatic policy construction, provided as an embodiment of the present invention. Figure 1 As shown, the system includes a business configuration layer, a management application service layer, a policy auto-build layer, a Kafka Connect execution layer, a Hook execution layer, a task orchestration and lifecycle management module, a monitoring and auditing module, and a task definition storage module. These modules collaborate using a unified task object, enabling business-level configurations to be automatically converted into executable source, sink, and hook tasks.
[0045] The business configuration terminal receives task description information input by the user. It can provide a graphical user interface (GUI) or a command-line interface (CLI) for business personnel to input task description information. This task description information includes at least the source database type, source table name, target database type, and target table name, and may also include business-level attributes for data synchronization, such as synchronization frequency and filtering conditions. The business configuration terminal sends the user-input task description information to the management application service layer in a standardized format.
[0046] The management application service layer communicates with the business configuration terminal to receive the task description information and construct a unified task object. The management application service layer is the core control plane of the system, responsible for parsing the task requests sent by the business configuration terminal, constructing standardized task objects according to the unified task description model, and sending the task objects to the policy auto-construction layer and the task definition storage module respectively.
[0047] The automatic strategy construction layer communicates with the management application service layer. It determines the corresponding strategy template based on the roles and data source types of each connector in the unified task object, maps the business-level input parameters in the task description information to the strategy template, and automatically generates the underlying execution parameters. The automatic strategy construction layer is the core module for implementing configuration abstraction in this invention; its specific working principle will be explained later. Figure 4 Provide a detailed description.
[0048] The Kafka Connect execution layer communicates with the policy auto-build layer to receive execution parameters from the underlying layer and execute standard data synchronization tasks. The Kafka Connect execution layer can be one or more Kafka Connect clusters, each running multiple worker processes responsible for executing specific connector tasks. The Kafka Connect execution layer receives connector configurations from the policy auto-build layer via its REST API, creating or updating corresponding source and sink connectors.
[0049] The Hook execution layer is used to execute extended processing tasks. It is an execution environment independent of the Kafka Connect execution layer and can be a custom task execution thread pool or process pool managed by the management application service layer. The Hook execution layer retrieves messages by subscribing to Kafka topics and executes HTTP delivery or other custom extended logic according to the hook configuration.
[0050] The task orchestration and lifecycle management module communicates with the management application service layer, the KafkaConnect execution layer, and the Hook execution layer, respectively, and is used to perform unified creation, start, pause, deletion, status query, and configuration editing operations on standard synchronous tasks and extended tasks.
[0051] The task definition storage module communicates with the management application service layer and is used to store the unified task objects. The task definition storage module can be a relational database, a non-relational database, or a file system, and is used to persistently store the definition information of all tasks so that the task state can be restored after a system restart.
[0052] The monitoring and auditing module records the operation logs and running logs of all tasks in the system, including operation records such as task creation, startup, pause, deletion, and configuration changes, as well as the running status and error information of the tasks, which facilitates the troubleshooting and auditing of maintenance personnel.
[0053] Figure 2 This is a schematic diagram of the unified task description model according to an embodiment of the present invention. Figure 2As shown, the unified task object includes at least a task name (taskName), a set of connectors (connectors), and a set of extended tasks (hookConfigs).
[0054] The connector set (connectors) is used to describe standard synchronization tasks. Each connector object includes at least the following fields: Name: The unique identifier of the connector.
[0055] Role: Used to identify whether the connector is a source role or a sink role.
[0056] Data source type (dataSource): Used to identify the data source type corresponding to the connector, such as SQL Server, MySQL, PostgreSQL, MongoDB, Elasticsearch, etc.
[0057] State: Indicates the current running status of the connector, such as creating, running, paused, stopped, or abnormal.
[0058] Common parameter set: Used to hold standard connection parameters, including but not limited to hostname, port, username, password, database name, table name, topic prefix, and task concurrency limit.
[0059] Enhanced parameter set (options): Used to hold connector-specific or enhanced configurations, such as snapshot mode, deletion policy (tombstones.on.delete, delete.on.null.values), replacement policy (replace.one.document), and schema change switch (include.schema.changes).
[0060] Furthermore, the enhanced parameter set `options` includes a `fields` substructure, serving as a carrier for front-end custom attribute information. This area does not require business personnel to understand the underlying connector semantics; instead, it allows the front-end to carry business tags, extended metadata, route identifiers, or other additional attributes beyond standard synchronous tasks. The custom attributes in `fields` can be read and used in subsequent template construction, hook processing, interface delivery, audit logging, or downstream extended logic, thus preserving flexible extension injection capabilities for the business side without disrupting the standard connection parameter structure.
[0061] The extended task collection (hookConfigs) is used to describe non-standard extended processing tasks. Each hook configuration includes at least a list of topics, a consumer group identifier (groupId), a task state (state), a request body template (body), a target HTTP address (httpUrl), and request header parameters (httpHeaders).
[0062] Figure 3 This is a flowchart illustrating the task creation and execution logic of an embodiment of the present invention. Figure 3 As shown, the task creation and execution process includes the following steps: First, business personnel input task description information through the business configuration terminal. The task description information expresses the business intent of "listening to which table in which database and synchronizing to which table in which database", and can selectively configure the hook extended processing method.
[0063] Secondly, the management application service layer constructs a unified task object based on the task description information. During the construction process, the management application service layer encapsulates the source database information into a connector object with the role of "source", encapsulates the target database information into a connector object with the role of "sink", and encapsulates the user-configured extended processing information into a "hookConfigs" object.
[0064] Then, the automatic policy construction layer determines the policy category for each connector based on its role and dataSource. For the source role, the system selects the source policy template corresponding to the source database; for the sink role, the system selects the sink policy template corresponding to the target database; and for hook configurations, the system selects the hook execution template. Subsequently, the system maps business-level input parameters to the policy templates, automatically completes default values, generates topic names, constructs the source-side listening scope, generates target-side write parameters, and pre-defines and optimizes configurations such as deletion policies, snapshot modes, replacement policies, and task concurrency.
[0065] Next, the task orchestration and lifecycle management module automatically builds the underlying execution parameters generated by the strategy layer, creating or updating the corresponding source and sink tasks in the Kafka Connect execution layer. The source task listens for or fetches changed data from a specified table in a specified database and writes the data to a Kafka topic; the sink task reads data from the topic and writes it to the target database, forming a complete synchronous link from the source database to the message topic and then to the target database.
[0066] Meanwhile, the task orchestration and lifecycle management module creates corresponding hook tasks in the Hook execution layer based on the hookConfigs in the unified task object. The Hook execution layer subscribes to a specified topic, maps the key, value, topic, timestamp, and other data in the message to the request body template according to the template variable replacement rules, constructs the HTTP request body and delivers it to the external business interface, or triggers other custom extended logic.
[0067] Finally, during operation, the task orchestration and lifecycle management module performs control operations such as creation, startup, pause, deletion, status query, and configuration editing on source tasks, sink tasks, and hook tasks through a unified lifecycle interface, and synchronizes the task status back to the unified control plane.
[0068] Figure 4 This is a schematic diagram illustrating the automatic strategy construction process according to an embodiment of the present invention. Figure 4 As shown, the workflow of the automatic strategy construction layer specifically includes the following steps: First, the strategy auto-build layer extracts the role and data source type of each connector from the unified task object.
[0069] Specifically, the strategy's automatic construction layer iterates through the connectors collection in the unified task object, extracting the role and dataSource fields from each connector object. The role field takes the value "source" or "sink", and the dataSource field takes the value of a specific database type identifier, such as "mysql", "postgresql", "sqlserver", "mongodb", etc.
[0070] Then, based on the role and the data source type, the corresponding strategy template is matched from the strategy template storage unit.
[0071] The strategy template storage unit pre-stores strategy templates corresponding to different data source types and roles. For example, for a scenario where role=source and dataSource=mysql, a MySQL Source strategy template is stored; for a scenario where role=sink and dataSource=mongodb, a MongoDB Sink strategy template is stored. Each strategy template defines the name, type, default value, and dependencies and constraints between all the underlying configuration parameters required by that type of connector.
[0072] Next, the business-level parameters in the common parameter set are mapped to the corresponding fields in the strategy template.
[0073] Specifically, the automatic policy construction layer reads the `common` field of the connector from the unified task object, extracts business-level parameters such as host address, port, username, password, database name, table name, topic prefix, and task concurrency, and assigns the values of these parameters to the corresponding configuration items in the policy template. For example, `common.hostname` is mapped to the `database.hostname` configuration item in the policy template, and `common.tableName` is mapped to the `table.include.list` configuration item in the policy template.
[0074] In addition, based on the data source type and the role, default values are automatically filled in to generate a topic name and construct the source-side listening range or the target-side write parameters.
[0075] For parameters not explicitly specified in the common parameters, the strategy auto-construction layer automatically fills in appropriate default values based on the data source type and role. For example, for the JDBC Source connector, it automatically sets appropriate parameters such as polling interval (poll.interval.ms) and batch size (batch.max.rows). Simultaneously, the strategy auto-construction layer automatically generates complete topic names based on the topic prefix and table name in the common parameters, ensuring that source and sink tasks use the same topic for data exchange.
[0076] Finally, the deletion strategy, snapshot mode, replacement strategy, and task concurrency are predefined and optimized to generate complete underlying execution parameters.
[0077] The strategy auto-build layer predefines and optimizes connector enhancement parameters based on data source type and business scenario. For example, it automatically enables appropriate snapshot modes for CDC scenarios; automatically configures tombstone message handling strategies for deletion operations; and automatically optimizes task concurrency (tasks.max) based on source data volume and target write capacity. Finally, the strategy auto-build layer outputs a complete set of connector configuration parameters that can be directly submitted to the Kafka Connect execution layer.
[0078] When a new database type needs to be connected, simply extend the corresponding data source strategy template in the strategy template storage unit. The unified task model and the above-mentioned automatic construction process can be used to support the new data source. Therefore, the system has the ability to inherit and extend data sources within the scope supported by Kafka Connect.
[0079] Figure 5 This is a schematic diagram illustrating the execution flow of a Hook extended task according to an embodiment of the present invention. Figure 5 As shown, the workflow of the Hook execution layer specifically includes the following steps: First, the topic list, consumer group identifier, request body template, target HTTP address, and request header parameters are read from the hookConfigs of the unified task object.
[0080] The Hook execution layer extracts hookConfigs configuration from the unified task object when the task is created. hookConfigs includes at least: topics (a list of Kafka topics to subscribe to), groupId (consumer group identifier, used for offset management), body (request body template, supporting variable placeholders), httpUrl (target HTTP interface address), and httpHeaders (HTTP request header parameters).
[0081] Then, the Hook execution layer subscribes to the specified Kafka topic and retrieves messages.
[0082] The Hook execution layer acts as a Kafka consumer, subscribing to topics in the `topics` using the `groupId` specified in `hookConfigs`. When a new message arrives, the Hook execution layer pulls the message from Kafka, obtaining complete information such as the message's key, value, topic, partition, and timestamp.
[0083] Next, the keys, values, topics, and timestamps in the message are mapped to the request body template according to the template variable substitution rules to construct the HTTP request body.
[0084] The Hook execution layer performs variable substitution on the request body template. The template variable substitution rules support referencing various fields of the message in the template using specific syntax (such as {{.key}}, {{.value}}, {{.topic}}, {{.timestamp}}, etc.). The Hook execution layer then replaces the actual data from the message into the corresponding positions in the template, generating a complete HTTP request body.
[0085] Finally, the constructed HTTP request body is delivered to the external business interface or triggers custom extension logic.
[0086] The Hook execution layer constructs an HTTP request using httpUrl and httpHeaders, and sends the request body generated in step S53 to the target HTTP interface. After delivery, the Hook execution layer records the delivery result (success / failure) and decides whether to retry based on the configuration. In addition to HTTP interface delivery, the Hook execution layer can also be replaced with message redelivery, file writing, cache writing, or other custom processing methods.
[0087] Unlike standard synchronous tasks, hook tasks do not rely on the sink implementation provided by Kafka Connect; instead, they are executed by the hook execution layer. However, in terms of management, hook tasks share the same control entry point as source and sink tasks, allowing operations such as creating, pausing, deleting, and editing task content.
[0088] Figure 6 This is a schematic diagram of a lifecycle management process according to an embodiment of the present invention. Figure 6 As shown, the task orchestration and lifecycle management module provides a unified task lifecycle interface, which can perform control operations such as creation, start, pause, deletion, status query, and configuration editing on standard synchronous tasks and hook tasks.
[0089] Specifically, the task orchestration and lifecycle management module includes the following units: The task creation unit is used to create or update corresponding source and sink tasks in the Kafka Connect execution layer based on the unified task object, and to create corresponding hook tasks in the Hook execution layer. For source and sink tasks, the task creation unit submits the connector configuration by calling the Kafka Connect REST API (PUT / connectors / {name} / config); for hook tasks, the task creation unit registers a new hook consumer instance in the Hook execution layer.
[0090] The task status management unit is used to obtain the running status of source and sink tasks via the REST API of the Kafka Connect execution layer (GET / connectors / {name} / status), and to obtain the running status of hook tasks, and synchronize the running status back to the unified control plane. The task status management unit periodically polls the status information of each execution layer and updates it to the task definition storage module to ensure consistency between the control plane and the execution plane.
[0091] The task control unit is used to start, pause, delete, and update configurations for source, sink, and hook tasks. For source and sink tasks, the task control unit performs the corresponding operations through the Kafka Connect REST API (PUT / connectors / {name} / stop is used to pause, and DELETE / connectors / {name} is used to delete). For hook tasks, the task control unit controls the lifecycle of hook consumers through an internal interface.
[0092] Specifically, for hook tasks, in addition to status control, the task control unit also supports editing and re-issuing task content such as request templates, target addresses, request headers, and custom attribute mappings; for source and sink tasks, the task control unit can resubmit the updated strategy results to the Kafka Connect execution layer to complete configuration updates and status write-back.
[0093] Example 2 This embodiment provides a data synchronization management method based on configuration abstraction and automatic policy construction, which is applied to the system described in Embodiment 1. Figure 7 This is a flowchart illustrating a data synchronization management method based on configuration abstraction and automatic policy construction, according to an embodiment of the present invention.
[0094] Please see Figure 7 The method includes the following steps S1 to S8: S1, Receive task request.
[0095] The task request is initiated by business personnel through the business configuration terminal. The task request contains task description information for data synchronization, including at least: source database type (such as SQL Server, MySQL, PostgreSQL, etc.), source database name, source table name, target database type (such as MongoDB, Elasticsearch, etc.), target database name, and target table name.
[0096] In addition, task requests can also include business-level attributes, such as synchronization frequency, filtering conditions, and whether to enable HTTP extended delivery and other extended processing configurations.
[0097] exist Figure 7 In step S1, the task description information input by the user is sent to the management application service layer in a unified format through the business configuration terminal. After receiving the information, the process proceeds to step S2.
[0098] S2, Analyzing the unified model.
[0099] After receiving the task request sent in step S1, the management application service layer constructs a unified task object based on the task description information, that is, parses and generates a unified task description model. The unified task object includes at least a connector set (connectors) and an extended task set (hookConfigs).
[0100] The connector set is used to describe standard data synchronization tasks. Each connector object includes at least: role (identifying whether the connector is a source or sink role), data source type (such as SQL Server, MongoDB, etc.), common parameter set (including host address, port, username, password, database name, table name, etc.), and enhanced parameter set options (including snapshot mode, deletion strategy, replacement strategy, etc.).
[0101] The extended task collection hookConfigs is used to describe non-standard extended processing tasks. Each hook configuration includes at least a list of topics, a consumer group identifier groupId, a request body template body, a target HTTP address httpUrl, and request header parameters httpHeaders.
[0102] For the specific structure of the unified task object, please refer to Figure 2 After the management application service layer completes the construction of the unified task object, it sends the unified task object to the policy auto-construction layer and the task definition storage module respectively.
[0103] exist Figure 7 In step S2, the management application service layer identifies and distinguishes between connectors (standard synchronous tasks) and hookConfigs (extended tasks) in the unified task object, preparing for the subsequent traffic splitting processing in steps S4, S5, and S6. After parsing is completed, the process proceeds to step S3.
[0104] S3, Automatic Strategy Construction.
[0105] The strategy auto-construction layer automatically constructs the underlying execution parameters based on the unified task object generated in step S2.
[0106] For a detailed breakdown of this step, please refer to [link / document name]. Figure 4 .like Figure 4 As shown, step S3 further includes the following sub-steps S31 to S35: S31. The strategy automatic construction layer extracts the role and data source type of each connector from the unified task object.
[0107] The strategy automatic construction layer traverses the connectors collection in the unified task object and extracts the role field (with the value "source" or "sink") and dataSource field (with the value being a specific database type identifier, such as "mysql", "postgresql", "sqlserver", "mongodb", etc.) for each connector object.
[0108] S32. Match the corresponding strategy template from the strategy template storage unit according to the role and the data source type dataSource.
[0109] The policy template storage unit pre-stores policy templates corresponding to different data source types and roles. For example, for a scenario where role=source and dataSource=sqlserver, the SQL Server Source policy template is matched; for a scenario where role=sink and dataSource=mongodb, the MongoDB Sink policy template is matched. Each policy template defines the name, type, default value, and dependencies and constraints between all the underlying configuration parameters required by that type of connector.
[0110] S33. Map the business-level parameters in the common parameter set to the corresponding fields of the strategy template.
[0111] Taking SQL Server Source as an example, the strategy auto-build layer maps the business parameters in common to the specific configuration parameters of the KafkaConnect JDBC Source connector, such as mapping common.hostname to the host address in connection.url, and common.tableName to table.include.list, etc. At the same time, it automatically sets technical parameters such as connector class name (connector.class) and value converter (value.converter).
[0112] S34. Automatically complete the default values according to the data source type and the role, generate the topic name, and construct the source end listening range or target end writing parameters.
[0113] For parameters not explicitly specified in the common parameters, the strategy auto-construction layer automatically fills in appropriate default values based on the data source type and role. For example, for the JDBC Source connector, it automatically sets appropriate parameters such as polling interval (poll.interval.ms) and batch size (batch.max.rows). Simultaneously, the strategy auto-construction layer automatically generates complete topic names based on the topic prefix and table name in the common parameters, ensuring that source and sink tasks use the same topic for data exchange.
[0114] S35. Predefined optimizations are performed on deletion strategies, snapshot modes, replacement strategies, and task concurrency to generate complete underlying execution parameters.
[0115] The strategy auto-build layer predefines and optimizes connector enhancement parameters based on data source type and business scenario. For example, it automatically enables appropriate snapshot modes for CDC scenarios; automatically configures tombstone message handling strategies for deletion operations; and automatically optimizes task concurrency (tasks.max) based on source data volume and target write capacity. Finally, the strategy auto-build layer outputs a complete set of connector configuration parameters that can be directly submitted to the Kafka Connect execution layer.
[0116] exist Figure 7 In step S3, the strategy-based automatic construction layer generates corresponding source and sink parameters for each connector in the unified task object, and also generates the parameters required for hook execution for hookConfigs. After the automatic construction is completed, the process proceeds to step S4.
[0117] S4. Issue the Source task.
[0118] The task orchestration and lifecycle management module automatically builds the underlying execution parameters of the source generated by the strategy in step S3, and creates or updates the corresponding source tasks in the Kafka Connect execution layer.
[0119] For a detailed breakdown of this step, please refer to [link / document name]. Figure 3 .like Figure 3 As shown, step S4 further includes the following sub-steps S41 to S42: S41, the task orchestration and lifecycle management module submits the source connector configuration through the Kafka Connect REST API.
[0120] The task orchestration and lifecycle management module calls the Kafka Connect REST API (PUT / connectors / {source_name} / config) to submit the source connector configuration parameters generated by the strategy auto-build layer to the Kafka Connect execution layer.
[0121] S42, the Kafka Connect execution layer starts the source task based on this configuration, establishing a source-side listening or change capture link.
[0122] After receiving the configuration, the Kafka Connect execution layer creates or updates the corresponding source connector instance and starts the source task. The source task listens for or fetches changed data (CDC) from a specified table in a specified database and writes the changed data to a Kafka topic.
[0123] exist Figure 7 In step S4, after the source task is issued, the process proceeds to step S5.
[0124] S5. Issue the Sink task.
[0125] The task orchestration and lifecycle management module automatically builds the underlying sink execution parameters generated by the strategy in step S3, and creates or updates the corresponding sink tasks in the Kafka Connect execution layer.
[0126] For a more detailed breakdown of this step, please refer to [link / reference]. Figure 3 .like Figure 3 As shown, step S5 further includes the following sub-steps S51 to S53: The S51 task orchestration and lifecycle management module submits sink connector configurations via the Kafka Connect REST API.
[0127] The task orchestration and lifecycle management module calls the Kafka Connect REST API (PUT / connectors / {sink_name} / config) to submit the sink connector configuration parameters generated by the strategy auto-build layer to the Kafka Connect execution layer.
[0128] The S52 and Kafka Connect execution layer start the sink task based on this configuration.
[0129] After receiving the configuration, the Kafka Connect execution layer creates or updates the corresponding sink connector instance and starts the sink task. The sink task reads data from the corresponding Kafka topic and writes it to the target database.
[0130] S53. Bind the topic and the target table to form a complete synchronization link from the source database to the message topic and then to the target database.
[0131] The topic name generated in step S34 by the automatic strategy construction layer is configured in both the "topics" output of the source task and the "topics" input of the sink task, ensuring that both use the same topic for data exchange. The source task writes the changed data from the source database to this topic, and the sink task reads data from the same topic and writes it to the target database, thereby completing the binding between the topic and the target table and forming a complete synchronization link.
[0132] exist Figure 7 In step S5, after the sink task is issued and the topic is bound, the process proceeds to step S6.
[0133] S6. Register Hook task.
[0134] The task orchestration and lifecycle management module creates corresponding hook tasks in the Hook execution layer and registers hook consumer instances based on the hookConfigs in the unified task object in step S2.
[0135] For a detailed breakdown of this step, please refer to [link / document name]. Figure 5 .like Figure 5 As shown, step S6 further includes the following sub-steps S61 to S64: S61. Read the topic list, consumer group identifier, request body template, target HTTP address and request header parameters from the hookConfigs of the unified task object.
[0136] The Hook execution layer extracts hookConfigs configuration from the unified task object when the task is created. hookConfigs includes at least: topics (a list of Kafka topics to subscribe to), groupId (consumer group identifier, used for offset management), body (request body template, supporting variable placeholders), httpUrl (target HTTP interface address), and httpHeaders (HTTP request header parameters).
[0137] S62. The Hook execution layer subscribes to the specified Kafka topic and retrieves messages.
[0138] The Hook execution layer acts as a Kafka consumer, subscribing to topics in the `topics` using the `groupId` specified in `hookConfigs`. When a new message arrives, the Hook execution layer pulls the message from Kafka, obtaining complete information such as the message's key, value, topic, partition, and timestamp.
[0139] S63. Construct the HTTP request body according to the template.
[0140] The Hook execution layer performs variable substitution on the request body template, mapping the keys, values, topics, and timestamps in the message to the request body template according to the template variable substitution rules to construct the HTTP request body. The template variable substitution rules support referencing various fields of the message in the template using specific syntax (such as {{.key}}, {{.value}}, {{.topic}}, {{.timestamp}}, etc.). The Hook execution layer then replaces the actual data from the message in the corresponding positions in the template, generating a complete HTTP request body.
[0141] S64. Deliver the constructed HTTP request body to the external business interface or trigger custom extension logic.
[0142] The Hook execution layer constructs an HTTP request using httpUrl and httpHeaders, and sends the request body generated in step S63 to the target HTTP interface. After delivery, the Hook execution layer records the delivery result (success / failure) and decides whether to retry based on the configuration.
[0143] Unlike standard synchronization tasks, hook tasks do not rely on the sink implementation that comes with Kafka Connect; instead, they are executed by the Hook execution layer.
[0144] exist Figure 7 In step S6, after registering the hook task, the process proceeds to step S7.
[0145] S7, Running state control.
[0146] The task orchestration and lifecycle management module performs unified runtime control operations on the source and sink tasks created in steps S4 and S5, as well as the hook tasks created in step S6.
[0147] For a detailed breakdown of this step, please refer to [link / document name]. Figure 6 .like Figure 6 As shown, step S7 further includes the following sub-steps S71 to S74: S71. The task creation unit creates or updates the corresponding source task and sink task in the Kafka Connect execution layer according to the unified task object, and creates the corresponding hook task in the Hook execution layer.
[0148] For source and sink tasks, the task creation unit submits the connector configuration by calling the Kafka Connect REST API (PUT / connectors / {name} / config); for hook tasks, the task creation unit registers a new hook consumer instance in the hook execution layer.
[0149] S72, the task control unit performs start, pause, delete, and configuration update operations on source tasks, sink tasks, and hook tasks.
[0150] For source and sink tasks, the task control unit performs corresponding operations through the Kafka Connect REST API (PUT / connectors / {name} / stop is used to pause, DELETE / connectors / {name} is used to delete); for hook tasks, the task control unit controls the lifecycle of hook consumers through internal interfaces.
[0151] S73. Configuration editing operations for source and sink tasks include: resubmitting the updated strategy results to the Kafka Connect execution layer to complete the configuration update and status write-back.
[0152] S74. The configuration editing operations for hook tasks include: editing and re-issuing the request template, target address, request headers, and custom attribute mappings.
[0153] exist Figure 7 In step S7, after completing the running state control, the process proceeds to step S8.
[0154] S8. Monitoring and backtracking.
[0155] The task orchestration and lifecycle management module performs unified monitoring and backtracking operations on the source and sink tasks created in steps S4 and S5, as well as the hook tasks created in step S6.
[0156] This step further includes the following sub-steps S81 to S84: S81. The task status management unit obtains the running status of the source task and sink task through the REST API of the Kafka Connect execution layer (GET / connectors / {name} / status), and obtains the running status of the hook task through the internal interface, and synchronizes the running status back to the unified control plane.
[0157] The task status management unit periodically polls the status information of each execution layer and updates it to the task definition storage module to ensure the consistency of the status between the control plane and the execution plane.
[0158] S82. The monitoring and auditing module records the operation logs and running logs of all tasks in the system, including operation records such as task creation, startup, pause, deletion, and configuration changes, as well as the running status and error information of the tasks.
[0159] S83. The monitoring and auditing module executes a retry policy for failed hook tasks and determines the number of retries and the retry interval based on the configuration.
[0160] S84. All task status information, log information, and audit information are displayed uniformly on the management interface. Users can view the running status and historical records of each task through the unified management interface, which facilitates troubleshooting and audit tracing by operation and maintenance personnel.
[0161] Example 3 This embodiment uses an enterprise data integration project as an example to provide typical application scenarios of the system and method in Embodiment 1 and Embodiment 2, in order to further illustrate the technical solution of the present invention.
[0162] In an enterprise data integration project, business personnel need to synchronize customer information (stored in the dbo.Customer table of the SQL Server database) from the core business system to the analytical database (the customer collection of MongoDB) in real time. At the same time, they need to notify the customer relationship management (CRM) system of every change in customer information via an HTTP interface.
[0163] The operation process for business personnel using the data synchronization task management system provided by this invention is as follows: The first step is to log in to the business configuration terminal and select "Create Data Synchronization Task".
[0164] The second step is to select "SQL Server" as the source in the task configuration interface, fill in the database connection address, port, authentication information, and select the source table "dbo.Customer".
[0165] The third step is to select "MongoDB" as the target, enter the MongoDB connection address, and select the target collection "customer".
[0166] Fourth, in the "Extended Processing" area, select "HTTP Delivery," enter the HTTP interface address of the CRM system, and configure the fields to be delivered in the request body template, for example: json { "customerId": "{{.value.id}}", "customerName": "{{.value.name}}", "phone": "{{.value.phone}}", "operation": "{{.value.op}}", "timestamp": "{{.timestamp}}" } Fifth step, click "Create Task".
[0167] The system will automatically perform the following operations: Construct a unified task object, which includes a connector with role as source and dataSource as SQL Server, a connector with role as sink and dataSource as MongoDB, and a hookConfigs that points to the CRM system's HTTP interface.
[0168] The automatic strategy building layer matches the SQL Server Source strategy template and the MongoDB Sink strategy template respectively, and automatically generates complete Kafka Connect connector configuration parameters.
[0169] The task orchestration and lifecycle management module creates SQL Server Source connectors and MongoDB Sink connectors in the Kafka Connect execution layer, establishing a synchronization link from SQL Server to MongoDB.
[0170] The task orchestration and lifecycle management module creates HTTP delivery tasks at the Hook execution layer and subscribes to the corresponding Kafka topics.
[0171] When data changes occur in the Customer table in SQL Server, the source task captures the changes and writes them to a Kafka topic; the sink task reads data from the topic and writes it to MongoDB; the hook task simultaneously reads data from the topic, assembles an HTTP request according to a template, and delivers it to the CRM system.
[0172] Throughout the process, business users do not need to access any of the underlying configuration parameters of Kafka Connect. They only need to express their business intent to complete the configuration of complex data synchronization and extended processing links.
[0173] Example 4 This invention also provides an electronic device, including: a processor, a transmitting device, an input device, an output device, and a memory. The processor may be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory may be implemented using read-only memory (ROM), static storage devices, dynamic storage devices, or random access memory (RAM), and is used to store computer program code. The computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device performs a method as described in any of the above possible implementations.
[0174] Example 5 The present invention also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor of an electronic device, cause the processor to perform a method as described in any of the possible implementations above. The computer-readable storage medium can be any medium capable of storing a computer program, including but not limited to magnetic storage devices (such as hard disks, floppy disks, magnetic tapes, etc.), optical storage devices (such as CD-ROMs, DVD-ROMs, etc.), and semiconductor storage devices (such as EEPROMs, flash memory, etc.).
[0175] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention.
Claims
1. A data synchronization management system based on configuration abstraction and automatic strategy construction, characterized in that, include: The business configuration end is used to receive task description information input by the user; The management application service layer communicates with the business configuration layer and is used to receive the task description information and construct a unified task object. The automatic strategy construction layer communicates with the management application service layer and is used to determine the corresponding strategy template according to the role and data source type of each connector in the unified task object, and map the business-level input parameters in the task description information to the strategy template to automatically generate the underlying execution parameters. The Kafka Connect execution layer communicates with the policy auto-construction layer to receive the underlying execution parameters and execute standard data synchronization tasks. Hook the execution layer to perform extended processing tasks; The task orchestration and lifecycle management module communicates with the management application service layer, the Kafka Connect execution layer, and the Hook execution layer, respectively, and is used to perform unified creation, start, pause, deletion, status query, and configuration editing operations on standard synchronization tasks and extended tasks. The task definition storage module is communicatively connected to the management terminal application service layer and is used to store the unified task object.
2. The system according to claim 1, characterized in that, The unified task object includes at least a task name, a connector set (connectors), and an extended task set (hookConfigs); wherein, the connector set (connectors) is used to describe standard synchronization tasks, and each connector object includes at least a name, a role, a data source type (dataSource), a state, a common parameter set (common), and an enhanced parameter set (options); the role is used to identify whether the connector is a source role or a target role.
3. The system according to claim 2, characterized in that, The common parameter set is used to accommodate standard connection parameters, including host address, port, username, password, database name, table name, topic prefix, and task concurrency. The options parameter set is used to accommodate connector-specific or enhanced configurations, including snapshot mode, deletion policy, replacement policy, and mode change switch.
4. The system according to claim 3, characterized in that, The enhanced parameter set options also includes a fields substructure, which serves as a carrier area for front-end custom attribute information, used to carry business tags, extended metadata, or route identifiers; the custom attributes in fields can be read and used in subsequent template construction, hook processing, interface delivery, audit logs, or downstream extended logic.
5. The system according to claim 1, characterized in that, The automatic strategy construction layer includes: The strategy template storage unit is used to store strategy templates corresponding to different data source types and different roles; The strategy matching unit is used to match the corresponding strategy template based on the role of the connector and the data source type dataSource in the unified task object. The parameter mapping unit is used to map the business-level input parameters in the task description information to the strategy template, automatically complete the default values, generate the topic name, construct the source end listening range, generate the target end writing parameters, and predefine and optimize the deletion strategy, snapshot mode, replacement strategy and task concurrency. The template extension unit is used to extend the corresponding data source strategy template when a new database type is connected.
6. The system according to claim 1, characterized in that, The Hook execution layer includes: The configuration reading unit is used to read the topic list, consumer group identifier, task status, request body template, target HTTP address and request header parameters from the hookConfigs of the unified task object; The message subscription unit is used to subscribe to a specified Kafka topic and retrieve messages; The template rendering unit is used to map the keys, values, topics, and timestamps in the message to the request body template according to the template variable replacement rules, thereby constructing the HTTP request body; The request delivery unit is used to deliver the constructed HTTP request body to an external business interface or trigger custom extension logic.
7. The system according to claim 1, characterized in that, The task orchestration and lifecycle management module includes: The task creation unit is used to create or update the corresponding source task and sink task in the Kafka Connect execution layer according to the unified task object, and to create the corresponding hook task in the Hook execution layer. The task status management unit is used to obtain the running status of source tasks and sink tasks through the REST API of the Kafka Connect execution layer, and to obtain the running status of hook tasks, and synchronize the running status back to the unified control plane. The task control unit is used to start, pause, delete, and update the configuration of source, sink, and hook tasks.
8. A data synchronization management method based on configuration abstraction and automatic policy construction, characterized in that, The method includes the following steps: S1. Receive a task request, wherein the task request contains task description information for data synchronization, including at least the source database type, source database name, source table name, target database type, target database name, and target table name; S2. The management application service layer constructs a unified task object based on the task description information. The unified task object includes at least a connector set (connectors) and an extended task set (hookConfigs). Each connector object includes at least a role, a data source type (dataSource), a common parameter set (common), and an enhanced parameter set (options). S3. The strategy automatic construction layer automatically constructs the underlying execution parameters based on the unified task object, generates corresponding source parameters and sink parameters for each connector in the unified task object, and generates the parameters required for hook execution for hookConfigs. S4. The task orchestration and lifecycle management module creates or updates the corresponding source task in the Kafka Connect execution layer based on the underlying execution parameters of the source, and establishes a source-end listening or change capture link. S5. The task orchestration and lifecycle management module creates or updates the corresponding sink task in the Kafka Connect execution layer according to the underlying sink execution parameters, binds the topic and the target table, and forms a complete synchronization link from the source database to the message topic and then to the target database. S6. The task orchestration and lifecycle management module creates a corresponding hook task in the Hook execution layer based on the hookConfigs in the unified task object, registers hook consumer instances, subscribes to specified topics, and constructs an HTTP request body according to the template to execute extended processing logic. S7, the task orchestration and lifecycle management module performs unified runtime control operations on source tasks, sink tasks, and hook tasks, including starting, pausing, deleting, and configuration editing; The S8 task orchestration and lifecycle management module performs unified monitoring and backtracking operations on source tasks, sink tasks, and hook tasks, records status, logs, and audit information, and implements retry strategies for tasks that fail to be delivered.
9. The method according to claim 8, characterized in that, Step S3 specifically includes: S31. The strategy automatic construction layer extracts the role and data source type (dataSource) of each connector from the unified task object; S32. Match the corresponding strategy template from the strategy template storage unit according to the role and the data source type dataSource; S33. Map the business-level parameters in the common parameter set to the corresponding fields of the strategy template; S34. Automatically fill in the default value according to the data source type and the role, generate the topic name, and construct the source end listening range or the target end writing parameters; S35. Predefined optimizations are performed on deletion strategies, snapshot modes, replacement strategies, and task concurrency to generate complete underlying execution parameters.
10. The method according to claim 8, characterized in that, Step S5 specifically includes: S51, the task orchestration and lifecycle management module submits the sink connector configuration through the Kafka Connect REST API; The S52 and Kafka Connect execution layer start the sink task according to this configuration; S53. Bind the topic and the target table to form a complete synchronization link from the source database to the message topic and then to the target database.
