A big data update method and system based on Kafka and FlinkCDC

Abstract

The present disclosure relates to the field of database technology and provides a big data update method and system based on Kafka and Flink CDC. The method includes a configuration module receiving data source environment configuration and establishing a connection channel; Flink CDC capturing and processing change data; a Kafka topic receiving update messages and writing them to a corresponding dedicated message queue based on message type; a Flink job management module processing the message queue; and a target database updating the big data based on the processing results. Change events captured by Flink CDC are distributed to independent Kafka topics by table type to achieve physical isolation, and are written to corresponding dedicated message queues by message type, achieving both physical isolation and traffic tiering control, ensuring that primary table data is processed first.

Description

Technical Field

[0001] The present disclosure belongs to the field of database technology, and in particular relates to a big data update method and system based on Kafka and Flink CDC. Background Art

[0002] In data warehouse synchronization scenarios for financial enterprises, frequent changes to master and sub-tables can lead to extremely high concurrent write pressure on downstream databases. Traditional Flink CDC-based synchronization systems lack a flow control mechanism, and master and sub-tables share the same message channel. Large-scale data updates can lead to resource contention, resulting in data update delays and an inability to prioritize important data in the master table.

[0003] In existing technologies, Kafka is often used to decouple data production and consumption, but its deep integration with Flink CDC and priority scheduling solutions are not yet mature.

[0004] In view of this, it is necessary to provide a big data update method and system based on Kafka and Flink CDC. Summary of the Invention

[0005] To solve the above problems, the present disclosure provides a big data update method and system based on Kafka and Flink CDC, which adopts Kafka layered buffer design to avoid resource competition and ensure that the main table data is processed first.

[0006] In a first aspect, the present disclosure provides a big data update method based on Kafka and Flink CDC, comprising:

[0007] Step S1: The configuration module receives the data source environment configuration and establishes a connection channel: it receives the data information defined by the user through the configuration template; parses and registers the data information and establishes a connection channel to the source database; the data information includes: the source database type and connection parameters, the target database type and address, and the type definition of each data table;

[0008] Step S2: Flink CDC receives the real-time data change stream from the data source and obtains changes to the master table and sub-tables through pre-configured capture components. It formats the changed data to generate master table update messages and sub-table update messages.

[0009] Step S3: The Kafka topic receives the master table update message and writes it to a pre-established master table-specific Kafka topic for storage and transmission, resulting in an asynchronous message queue for master table data changes. The master table-specific Kafka topic adopts a multi-partition high-throughput strategy. The topic also receives the sub-table update message and writes it to a pre-established sub-table-specific Kafka topic for storage and transmission, resulting in an asynchronous message queue for sub-table data changes. The sub-table-specific Kafka topic adopts a compact storage strategy.

[0010] Step S4: The Flink job management module processes the asynchronous message queues for master table data changes and the asynchronous message queues for sub-table data changes to obtain processing results.

[0011] Step S5: The target database updates the database using the processing result through the established connection channel.

[0012] Furthermore, the data information also includes: the priority configuration corresponding to each data table; the Flink job management module includes: the Flink JobManager priority weight calculation module, the high-priority task thread pool, and the low-priority task queue;

[0013] Step S4 specifically includes:

[0014] Step S41: The Flink JobManager receives the main table update message and sub-table update message from the Kafka topic, adds the main table update message and sub-table update message to the task queue for real-time processing, and obtains a list of update tasks to be scheduled based on the source table type and preset priority attributes.

[0015] Step S42: The priority weight calculation module calculates the corresponding priority weight value according to the table type, data freshness and business tag of each task, and adds the priority weight value to the task metadata to obtain a weighted task queue;

[0016] Step S43: The resource scheduling module receives the weighted task queue, sorts it according to its weight value, and performs a diversion process: high-priority tasks with high weight are extracted from the queue and assigned to the high-priority dedicated channel, while low-priority tasks with low weight are retained in the ordinary channel for processing, thus obtaining two types of task sequences distinguished by priority.

[0017] Step S44: The high-priority task thread pool receives the high-priority task and submits it to the dedicated thread pool for parallel execution, processing the key data update operation at high speed and obtaining the high-priority task execution result in real time;

[0018] Step S45: The low-priority task queue receives low-priority tasks, adds them to the shared waiting queue, and queues them for processing in a predetermined order. These tasks are executed sequentially when the system is idle or under low load, and the low-priority task execution results are obtained in sequence.

[0019] Furthermore, the target database includes multiple target database shards;

[0020] After step S4 and before step S5, the method further includes: the Flink job management module uses a window sharding algorithm to divide data batches according to event time windows, and shards the high-priority task execution results and the low-priority task execution results based on the primary key hash value, wherein the number of data shards is equal to the number of shards of the target database;

[0021] The step S5 specifically includes:

[0022] Each target database shard receives the high-priority task execution result or the low-priority task execution result, as well as the pre-commit instruction issued by the two-phase commit coordinator, writes the data update operation into its own transaction buffer for pre-commit processing, and obtains the pre-commit execution result;

[0023] The two-phase commit coordinator receives the pre-commit results returned by each target database shard. When it determines that the pre-commit of all shards has been successfully completed, it sends a global commit instruction to all target database shards to commit the transaction.

[0024] Furthermore, it also includes:

[0025] Step S6: The monitoring platform collects task indicators in real time and displays them dynamically through the dashboard. Failed tasks are automatically transferred to HDFS and trigger exponential backoff retries. At the same time, external triggers for exponential backoff retries are received. Task indicators include some or all of throughput, latency, and error rate.

[0026] Furthermore, a transaction ID traceability module is embedded in Kafka; the method also includes:

[0027] The transaction ID traceability module marks data change events with a globally unique serial number and implements end-to-end Exactly-Once semantics by comparing the target database's version snapshots.

[0028] Secondly, a big data update system based on Kafka and Flink CDC is provided, including: a configuration module, Flink CDC, Kafka Topic, Flink job management module, and target database;

[0029] The configuration module is used to receive data source environment configuration and establish a connection channel: it receives data information defined by the user through the configuration template; parses and registers the data information and establishes a connection channel to the source database; the data information includes: the source database type and connection parameters, the target database type and address, and the type definition of each data table;

[0030] Flink CDC receives real-time data change streams from data sources and uses pre-configured capture components to capture changes to the master table and sub-tables. It then formats the changed data to generate master table update messages and sub-table update messages.

[0031] Kafka Topic: receives master table update messages and writes them to a pre-established master table-specific KafkaTopic for storage and transmission, resulting in an asynchronous message queue for master table data changes. This master table-specific KafkaTopic adopts a multi-partition high-throughput strategy. It also receives sub-table update messages and writes them to a pre-established sub-table-specific KafkaTopic for storage and transmission, resulting in an asynchronous message queue for sub-table data changes. This sub-table-specific KafkaTopic adopts a compact storage strategy.

[0032] The Flink job management module processes the asynchronous message queues for master table data changes and sub-table data changes to obtain processing results.

[0033] The target database is used to update the database with the processing results through the established connection channel.

[0034] Furthermore, the data information also includes: priority configuration corresponding to each data table; the system also includes: a resource scheduling module; the Flink job management module includes: a Flink JobManager priority weight calculation module, a high-priority task thread pool, and a low-priority task queue;

[0035] The Flink JobManager receives primary and sub-table update messages from Kafka topics, adds them to the task queue for real-time processing, and obtains a list of update tasks to be scheduled based on their source table type and preset priority attributes.

[0036] The priority weight calculation module is used to calculate the corresponding priority weight value according to the table type, data freshness and business tag of each task, and append the priority weight value to the task metadata to obtain a weighted task queue;

[0037] The resource scheduling module is used to receive the weighted task queue, sort them according to their weight values, and perform diversion processing: high-priority tasks with high weights are extracted from the queue and assigned to high-priority dedicated channels, while low-priority tasks with low weights are retained in the ordinary channel waiting to be processed, resulting in two types of task sequences distinguished by priority;

[0038] The high-priority task thread pool is used to receive high-priority tasks and submit them to a dedicated thread pool for parallel execution, processing key data update operations at high speed and obtaining high-priority task execution results in real time;

[0039] The low-priority task queue is used to receive low-priority tasks, add them to the shared waiting queue and queue them for processing in a predetermined order, and execute these tasks in sequence when the system is idle or under low load, to obtain the low-priority task execution results processed sequentially.

[0040] Furthermore, it also includes: a two-phase commit coordinator; the target database includes multiple target database shards;

[0041] The Flink job management module also uses a window sharding algorithm to divide data batches into event time windows. It also uses the primary key hash value to shard the results of high-priority and low-priority tasks. The number of data shards is equal to the number of shards in the target database.

[0042] The target database shard receives the high-priority task execution results or low-priority task execution results, as well as the pre-commit instructions issued by the two-phase commit coordinator, writes the data update operation into its own transaction buffer for pre-commit processing, and obtains the pre-commit execution results;

[0043] The two-phase commit coordinator is used to receive the pre-commit results returned by each target database shard. When it is determined that the pre-commit of all shards has been successfully completed, it sends a global commit instruction to all target database shards to commit the transaction.

[0044] Furthermore, it also includes:

[0045] The monitoring platform is used to collect task metrics in real time and display them dynamically through a dashboard. Failed tasks are automatically transferred to HDFS and trigger exponential backoff retries. The platform also receives external triggers for exponential backoff retries. Task metrics include some or all of throughput, latency, and error rate.

[0046] Furthermore, it also includes: a transaction ID traceability module embedded in Kafka;

[0047] The transaction ID traceability module is used to mark data change events with globally unique serial numbers and implement end-to-end exactly-once semantics by comparing them with version snapshots of the target database.

[0048] Compared with the prior art, the present disclosure has the following advantages:

[0049] In the solution of this disclosed embodiment, change events captured by Flink CDC are distributed to independent Kafka topics by table type (master table / sub-table), achieving physical isolation. Differentiated partition and replica numbers are configured for each topic. The master table topic adopts a multi-partition high-throughput strategy, while the sub-table topics adopt a compact storage strategy. This achieves physical isolation and hierarchical traffic control, ensuring that master table data is processed first.

[0050] Other features and advantages of the present disclosure will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present disclosure. The purposes and other advantages of the present disclosure can be realized and obtained by the structures indicated in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 FIG1 shows a schematic diagram of a big data update method based on Kafka and Flink CDC according to an embodiment of the present disclosure;

[0053] Figure 2 A schematic diagram of a big data update system based on Kafka and Flink CDC according to an embodiment of the present disclosure is shown;

[0054] Figure 3 The second schematic diagram of a big data update method based on Kafka and Flink CDC according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0056] First, the core improvements of the technical solution disclosed in this disclosure are explained.

[0057] 1. Kafka layered buffer design:

[0058] Distribute the change events captured by Flink CDC to independent Kafka topics based on table type (master table / sub-table) to achieve physical isolation;

[0059] Differentiated partition and replica numbers are configured for each topic. The main table topic adopts a multi-partition high-throughput strategy, and the sub-table topic adopts a compact storage strategy.

[0060] 2. Priority dynamic scheduling:

[0061] A priority weight calculation module is embedded in the Flink JobManager to dynamically allocate computing resources based on table type, data freshness, and business importance.

[0062] The main table tasks are assigned to a high-priority thread pool, while the sub-table tasks use a low-priority queue, and resource isolation is achieved through the YARN CapacityScheduler.

[0063] 3. Distributed task splitting:

[0064] Introducing a window sharding algorithm to split a single large transaction into multiple micro-batch tasks and write them in parallel to different database shards;

[0065] The two-phase commit protocol (2PC) is used to ensure the atomicity of sharding tasks and avoid data inconsistency caused by partial success.

[0066] 4. Visual monitoring and fault tolerance:

[0067] Build a real-time monitoring dashboard based on Prometheus+Grafana to display indicators such as task throughput, latency, and error rate;

[0068] Failed tasks automatically enter the retry queue, support exponential backoff retry strategy, and provide a manual trigger interface.

[0069] Secondly, the solution of the embodiment of the present disclosure is explained using the update of financial business big data as an application scenario.

[0070] Figure 1 A schematic diagram of a big data update method based on Kafka and Flink CDC according to an embodiment of the present disclosure is provided, which includes the following steps S1 to S5:

[0071] Step S1: Receive data source environment configuration and establish a connection channel.

[0072] Receive data information defined by the user through the configuration template; parse and register the data information and establish a connection channel to the source database; the data information includes: source database type and connection parameters, target database type and address, and type definitions of each data table;

[0073] Step S2: Flink CDC captures the changed data and processes it.

[0074] Flink CDC receives real-time data change streams from data sources and uses pre-configured capture components to capture changes to the master table and sub-tables. It then formats the changed data to generate master table update messages and sub-table update messages.

[0075] Step S3: The Kafka topic receives the update message and writes it to the corresponding dedicated message queue according to the message type.

[0076] The Kafka topic receives master table update messages and writes them to a pre-established master table-specific Kafka topic for storage and transmission, resulting in an asynchronous message queue for master table data changes. This master table-specific Kafka topic uses a multi-partition high-throughput strategy. It also receives sub-table update messages and writes them to a pre-established sub-table-specific Kafka topic for storage and transmission, resulting in an asynchronous message queue for sub-table data changes. This sub-table-specific Kafka topic uses a compact storage strategy. High throughput means that the throughput is greater than the set value.

[0077] Step S4: The Flink job management module processes the message queue.

[0078] The Flink job management module processes the asynchronous message queues for master table data changes and sub-table data changes to obtain processing results.

[0079] Step S5: The target database updates the big data based on the processing results.

[0080] The target database updates the database using the processing results through the established connection channel.

[0081] Again, combined Figure 2 and Figure 3 The solution of the embodiment of the present disclosure is described in detail:

[0082] like Figure 2 Figure 1 shows a big data update architecture based on Kafka and Flink CDC, an embodiment of the present disclosure. This architecture includes a data source layer, a CDC capture layer, a Kafka message queue layer, a Flink job management layer, a YARN resource scheduling layer, a storage layer, and a monitoring and pre-fault tolerance layer. This architecture illustrates the flow of big data. Figure 3 The following is a flow chart of a method for updating big data based on Kafka and Flink CDC according to an embodiment of the present disclosure. Figure 2 and Figure 3 As shown, the big data update method based on Kafka and Flink CDC in the embodiment of the present disclosure includes the following steps:

[0083] Step A: Receive the data information defined by the user through the configuration template, including the source database type and connection parameters (such as a MySQL data source), the target database type and address (such as a Snowflake data warehouse), and the type definition of each data table (main table or sub-table) and the corresponding priority configuration; parse and register the data information, establish a connection channel to the source database, and obtain the configured data source environment.

[0084] Specifically, users can configure data sources (such as MySQL, Kafka), target libraries (such as Snowflake, Elasticsearch), table types (main table / sub-table) and priority rules in the management interface.

[0085] The big data update system automatically generates Flink SQL or DataStream code by parsing the configuration template and encapsulates it into an executable Flink Job (JAR package).

[0086] The above configuration is the configuration of the correspondence between the data source and the target library fields, the configuration of the main table and sub-table types, and the configuration related to the database connection.

[0087] Priority configuration assigns a "priority configuration" to each data table (main table / sub-table). This priority configuration may include: table type (main table / sub-table), static priority label (such as "transaction", "auxiliary data", etc.), and business importance level.

[0088] In step A, users define priority influencing factors such as type attributes (main table / sub-table), business tags (such as "transaction", "risk control", and "ESG") for each data table through configuration templates. This information will be used to calculate the comprehensive priority weight value in subsequent task scheduling.

[0089] The priority configuration in step A is the static information (such as table type and business tag) configured by the user for each data table, which serves as the input for weight calculation.

[0090] Step B: The Flink CDC acquisition module receives the real-time data change stream from the data source and obtains changes to the main table and sub-tables through pre-configured capture components. The Debezium MySQL Connector captures changes to the main table from the MySQL binlog, and Flink SQL CDC captures changes to sub-tables from incremental snapshots. The change data is formatted to obtain standardized data update messages.

[0091] For example, when a field in a record in the MySQL master table is updated, the Flink CDC collection module captures the corresponding binlog event, parses the old and new values ​​of the field, and encapsulates them into a master table update message for subsequent processing.

[0092] Step C: The master table Kafka topic receives the master table update message output by the Flink CDC collection module and writes the message to a pre-established master table dedicated Kafka topic for storage and transmission, thus obtaining an asynchronous message queue for master table data changes.

[0093] By pushing the main table update message to the main table Kafka Topic, the main table change event is decoupled and buffered from the subsequent processing process.

[0094] Step D: The sub-table Kafka topic receives the sub-table update message output by the Flink CDC collection module and writes the message to the corresponding sub-table dedicated Kafka topic queue, obtaining an asynchronous message queue for sub-table data changes.

[0095] Sub-table update messages are queued in the sub-table Kafka topic and consumed downstream as low-priority events, thus providing an independent buffer channel for subsequent processing.

[0096] Steps C and D above distribute change events to Kafka topics based on table type: events for the primary table are written to a high-priority topic (for example, priority_high, with 16 partitions and 3 replicas); events for child tables are written to a low-priority topic (for example, priority_low, with Zstandard compression enabled).

[0097] Steps A through D above distribute CDC change events for the master and sub-tables to independent Kafka topics. The master table topic uses a multi-partition, high-throughput design (number of partitions = source table QPS / 1000), and the sub-table topics use message compression (Snappy algorithm) to reduce storage costs, achieving physical isolation and tiered traffic control. This solves the problem of traditional solutions where the master and sub-tables share the same message channel, resulting in core data being blocked by low-priority tasks. This design, through tiered topics, avoids resource contention and ensures that data from the financial transaction master table is processed first. This has resulted in a sixfold increase in master table data throughput (measured from 20,000 records / second to 120,000 records / second) and a 40% reduction in sub-table storage costs.

[0098] Step E: The task scheduling module receives update messages for the master and sub-tables from the master and sub-table Kafka topics, adds these messages to the real-time task queue, and generates a list of scheduled update tasks based on their source table type and preset priority attributes. The task scheduling module encapsulates each data change message entering the system, including whether it represents master or sub-table data and its corresponding priority, laying the foundation for subsequent scheduling.

[0099] Step F: The priority weight calculation module receives the list of tasks to be scheduled provided by the task scheduling module, calculates the corresponding priority weight value based on the table type, data freshness and business tag of each task, and appends the priority weight value to the task metadata to obtain a weighted task queue.

[0100] For example, for update tasks from the main table, the priority weight calculation module gives them a higher weight value (such as weight 5) according to the configuration, and gives update tasks from the sub-table a lower weight value (such as weight 1) to reflect the higher priority of the main table task relative to the sub-table task.

[0101] The priority weight calculation module receives the task list provided by the task scheduling module and performs the weight calculation logic shown in Table 1 below on each task:

[0102] Table 1 Weight calculation logic table

[0103]

[0104] The above-mentioned priority weight calculation module is deeply integrated with the Flink job scheduling module. Task resource allocation (such as thread pool selection and task slot binding) is completely based on the weight value, ensuring that task processing behavior is consistent with business importance.

[0105] In this step F, a "priority weight" is calculated based on the priority configuration in step A and real-time data (e.g., timestamp). A unified weighting rule is used to output a value (e.g., 13). Step G: The resource scheduling module receives the weighted task queue output by the priority weight calculation module, sorts the queue by weight, and performs task diversion.

[0106] Resource allocation (Task Slot) dynamically allocates task resources (thread pool / scheduling slot) based on the weights calculated in step F, using the weights as the sorting criteria and resource scheduling basis. Specifically, high-priority tasks with high weights are extracted from the queue and assigned to a dedicated high-priority channel, while low-priority tasks with low weights are retained in the regular channel for processing, resulting in two task sequences differentiated by priority. Through the resource scheduling module's diversion, high-priority main table update tasks and low-priority sub-table update tasks are directed to different execution paths.

[0107] Step H: The high-priority task thread pool receives high-priority tasks assigned by the resource scheduling module and submits them to a dedicated thread pool for parallel execution, processing critical data updates at high speed and achieving real-time high-priority task execution results. Multiple threads in the high-priority thread pool can simultaneously handle different master table update events, ensuring that these critical updates are completed with minimal latency.

[0108] Step I: The low-priority task queue receives the low-priority tasks assigned by the resource scheduling module, adds the low-priority tasks to the shared waiting queue and queues them for processing in a predetermined order, and executes these tasks in sequence when the system is idle or under load to obtain the low-priority task execution results processed sequentially.

[0109] Through the queuing mechanism of the low-priority task queue, non-critical data updates such as sub-tables are executed one by one in an orderly manner, which not only ensures that these updates are eventually completed, but also avoids affecting the timely processing of high-priority tasks.

[0110] Specifically, you can create two resource pools in YARN: a high-priority pool that occupies 70% of cluster resources and only allows the submission of master table tasks; and a low-priority pool that occupies 30% of resources and is used for sub-table tasks.

[0111] Submit tasks using the Flink YARN Application mode and specify resource pool parameters. High-priority tasks (weight ≥ 10) are assigned to a dedicated thread pool (number of threads = number of CPU cores x 2). Low-priority tasks (weight < 10) are assigned to a shared queue, and their resource usage is limited (maximum 30% of cluster resources) using the YARN Capacity Scheduler.

[0112] Steps E through I above embed a priority weight calculation module in the Flink JobManager. This dynamically allocates TaskSlot resources based on table type (master / sub-table), data freshness (event timestamp), and business tags (such as "transaction" and "risk control"). Master table tasks exclusively occupy a high-priority thread pool, while sub-table tasks utilize elastic queues. This addresses the existing technical issue of fixed resource allocation, which fails to adapt to the volatile demands of financial services and results in insufficient resources for high-value tasks. The result is a reduction in core transaction data synchronization latency from 120ms to below 30ms, improving resource utilization by 50%.

[0113] The above steps E to I also implement the Flink job management module to process the asynchronous message queues of the main table data changes and the asynchronous message queues of the sub-table data changes to obtain processing results.

[0114] Step J: The Flink job management module uses a window sharding algorithm to divide data batches into event time windows. It then shards the execution results of high-priority and low-priority tasks based on the primary key hash value. The number of data shards is equal to the number of shards in the target database.

[0115] Step K: The Flink job management module writes the data shards to the shard controller, and the shard controller writes the data in parallel to the target database shards.

[0116] Step L: The target database shard 1 receives the task data processed by the high-priority task thread pool or the target database shard 1 and the pre-commit instruction issued by the two-phase commit coordinator, writes the data update operation into the transaction buffer of the target database shard L for pre-commit processing, and obtains the pre-commit execution result of shard L. This pre-commit operation belongs to the first stage of the two-phase commit protocol. The data update is recorded on shard L but is not formally submitted yet, waiting for the global coordination instruction. Step M: The target database shard 2 receives the update data contained in the pre-commit instruction sent by the two-phase commit coordinator, writes the update data into the transaction log of the target database shard M for pre-commit, and obtains the pre-commit execution result of shard M.

[0117] Target database shard 1 and target database shard 2 perform similar pre-commit steps, recording the temporary status of the updated data locally in preparation for the final commit.

[0118] Step N: Target database shard 3 receives the updated data contained in the pre-commit instruction sent by the two-phase commit coordinator, and performs the same pre-commit operation as steps L and M in shard N to obtain the pre-commit execution result of shard N.

[0119] By performing the same pre-commit on the target database shard 3, all related shards are put into a ready state, waiting for global commit or rollback instructions.

[0120] Step O: The two-phase commit coordinator receives the pre-commit execution results returned by target database shard 1, target database shard 2, and target database shard 3. If it determines that the pre-commit for all shards has completed successfully, it sends a global commit command to all target database shards to commit the transaction. If it detects that the pre-commit for any shard has failed, it sends a global rollback command.

[0121] The two-phase commit coordinator then completes the consistent commit of the entire data update transaction on each shard, obtaining the final update result confirmed by the target database. Through the two-phase commit coordination of the two-phase commit coordinator, multiple shards are guaranteed to either successfully apply the update or abandon the update altogether in the event of any failure, thus ensuring data consistency.

[0122] Step P: The target database receives the global commit instruction and finally confirms the update result.

[0123] The above steps L to P also implement the update of the target database.

[0124] Steps L to P above use a window sharding algorithm to divide data batches into event time windows and shard the data based on the primary key hash value (N = the number of target database shards). Each shard is written asynchronously through Flink Async I / O, and a two-phase commit protocol (2PC) is introduced to ensure the atomicity of sharded transactions. This solves the problem of traditional full writes easily causing database locks and single point failures leading to data inconsistency. The database lock rate can be reduced by 98%, and the write throughput reaches millions of seconds after the sharding task parallelism is improved.

[0125] Step Q: The monitoring platform receives the operating indicator data generated by each processing step from the task scheduling module to the two-phase submission coordinator, aggregates and analyzes the indicator data to obtain a key performance indicator information set, and provides the indicator set to the subsequent indicator storage module.

[0126] The indicators summarized by the monitoring platform include data throughput, processing delay, number of task successes / failures, etc., which are used to comprehensively monitor the operation status of the data update process.

[0127] Specifically, the master table task enables Flink Checkpointing (interval = 1 minute) to ensure state persistence; the sub-table task uses the Lazy State mode to reduce memory usage.

[0128] To improve monitoring accuracy and task reliability, the master table task uses Flink's Checkpointing mechanism (with a one-minute period) to ensure persistent monitoring and fault-tolerant readiness states and facilitate failure recovery. To reduce resource consumption, the child table tasks employ a lazy state management model, initializing state only when needed, thereby optimizing memory usage. This mechanism ensures high reliability for critical tasks and lightweight execution for secondary tasks, enabling the monitoring platform to collect key operational metrics while maintaining optimal performance.

[0129] Step R: The indicator storage module Prometheus receives the key performance indicator information output by the monitoring platform, stores the indicator data in a time series format, and obtains a historical performance indicator database that can be queried.

[0130] Specifically, task metrics can be pushed to Prometheus through Flink Metrics Reporter.

[0131] The indicator storage module uses Prometheus to store indicators persistently, providing data support for post-analysis and real-time query of system operation status.

[0132] Step S: The visual monitoring module Grafana extracts the required data from the historical performance indicator database of the indicator storage module, receives the corresponding indicator data set, visualizes the data, generates real-time updated monitoring charts and dashboards, and obtains an intuitive interface for displaying the running status of the data update process.

[0133] Grafana can display priority distribution heat maps and track the resource usage of high- and low-priority tasks in real time. Through the graphical interface of the visual monitoring module, operations personnel can view key metrics such as latency, throughput, and error rates of the data pipeline in real time.

[0134] Step T: The alarm notification module receives the abnormal alarm information sent by the monitoring platform, encapsulates the abnormal information into an alarm notification message according to a preset template, and sends it to the pre-configured enterprise WeChat or DingTalk notification channel to obtain an instant alarm notification for the abnormal event.

[0135] With the help of the notification mechanism of the alarm notification module, when errors occur in the data update process or performance indicators exceed the limit, the relevant person in charge can receive the alarm immediately and intervene in time.

[0136] Step U: The failed task retry queue receives task failure information from the two-phase commit coordinator or execution thread (high-priority task thread pool / low-priority task queue), adds the failed task and its associated data to the retry waiting queue, records the failure reason, and obtains a list of tasks to be retried.

[0137] When a data update task fails to complete successfully due to a write failure in the target database or other exceptions, the failed task retry queue registers the task for later retry.

[0138] Step V: The manual trigger module receives the re-execution instruction manually triggered by the user, extracts the corresponding failed task from the list of tasks to be retried in the failed task retry queue, and resubmits the task to the task scheduling module for reprocessing, thereby obtaining a task instance that re-enters the data update process, thereby realizing the secondary execution of the failed data update task.

[0139] Through the manual triggering mechanism of the manual triggering module, operation and maintenance personnel can intervene in abnormal tasks to ensure that the data update process ultimately completes the reliable implementation of all tasks.

[0140] In steps Q to V above, Prometheus can be integrated to collect task metrics (throughput, latency, and error rate) in real time, and dynamically displayed on the Grafana dashboard. Failed tasks are automatically transferred to HDFS and trigger exponential backoff retries (retry interval = 2^N minutes). Manual triggering is also supported. This solves the technical problems of existing systems lacking fine-grained monitoring and flexible fault tolerance, and relying on manual troubleshooting for fault recovery. This reduces operation and maintenance response time from hours to minutes, and achieves a success rate of over 95% for automatic task repair.

[0141] In addition, a transaction ID traceability module can be embedded in Kafka, and data change events can be marked with a globally unique serial number (Snowflake algorithm). Combined with the version snapshot comparison of the target database, end-to-end Exactly-Once semantics can be achieved. This can solve the problem of strict data consistency requirements in financial scenarios, where traditional solutions are difficult to avoid duplication or loss of data, and achieve a technical effect of reducing the incidence of data consistency problems from 0.1% to 0.001%, which meets the regulatory standards of the financial industry. Based on the same inventive concept as the above-disclosed content, the present disclosure also provides a big data update system based on Kafka and Flink CDC, including: a configuration module, Flink CDC, Kafka Topic, a Flink job management module, and a target database;

[0142] The configuration module is used to receive data source environment configuration and establish a connection channel: it receives data information defined by the user through the configuration template; parses and registers the data information and establishes a connection channel to the source database; the data information includes: the source database type and connection parameters, the target database type and address, and the type definition of each data table;

[0143] Flink CDC receives real-time data change streams from data sources and uses pre-configured capture components to capture changes to the master table and sub-tables. It then formats the changed data to generate master table update messages and sub-table update messages.

[0144] Kafka Topic: receives master table update messages and writes them to a pre-established master table-specific KafkaTopic for storage and transmission, resulting in an asynchronous message queue for master table data changes. This master table-specific KafkaTopic adopts a multi-partition high-throughput strategy. It also receives sub-table update messages and writes them to a pre-established sub-table-specific KafkaTopic for storage and transmission, resulting in an asynchronous message queue for sub-table data changes. This sub-table-specific KafkaTopic adopts a compact storage strategy.

[0145] The Flink job management module processes the asynchronous message queues for master table data changes and sub-table data changes to obtain processing results.

[0146] The target database is used to update the database with the processing results through the established connection channel.

[0147] Furthermore, the data information also includes: priority configuration corresponding to each data table; the system also includes: a resource scheduling module; the Flink job management module includes: a Flink JobManager priority weight calculation module, a high-priority task thread pool, and a low-priority task queue;

[0148] The Flink JobManager receives primary and sub-table update messages from Kafka topics, adds them to the task queue for real-time processing, and obtains a list of update tasks to be scheduled based on their source table type and preset priority attributes.

[0149] The priority weight calculation module is used to calculate the corresponding priority weight value according to the table type, data freshness and business tag of each task, and append the priority weight value to the task metadata to obtain a weighted task queue;

[0150] The resource scheduling module is used to receive the weighted task queue, sort them according to their weight values, and perform diversion processing: high-priority tasks with high weights are extracted from the queue and assigned to high-priority dedicated channels, while low-priority tasks with low weights are retained in the ordinary channel waiting to be processed, resulting in two types of task sequences distinguished by priority;

[0151] The high-priority task thread pool is used to receive high-priority tasks and submit them to a dedicated thread pool for parallel execution, processing key data update operations at high speed and obtaining high-priority task execution results in real time;

[0152] The low-priority task queue is used to receive low-priority tasks, add them to the shared waiting queue and queue them for processing in a predetermined order, and execute these tasks in sequence when the system is idle or under low load, to obtain the low-priority task execution results processed sequentially.

[0153] Furthermore, the system further comprises: a two-phase commit coordinator; the target database comprises a plurality of target database shards;

[0154] The Flink job management module also uses a window sharding algorithm to divide data batches into event time windows. It also uses the primary key hash value to shard the results of high-priority and low-priority tasks. The number of data shards is equal to the number of shards in the target database.

[0155] The target database shard receives the high-priority task execution results or low-priority task execution results, as well as the pre-commit instructions issued by the two-phase commit coordinator, writes the data update operation into its own transaction buffer for pre-commit processing, and obtains the pre-commit execution results;

[0156] The two-phase commit coordinator is used to receive the pre-commit results returned by each target database shard. When it is determined that the pre-commit of all shards has been successfully completed, it sends a global commit instruction to all target database shards to commit the transaction.

[0157] Furthermore, it also includes:

[0158] The monitoring platform is used to collect task metrics in real time and display them dynamically through a dashboard. Failed tasks are automatically transferred to HDFS and trigger exponential backoff retries. The platform also receives external triggers for exponential backoff retries. Task metrics include some or all of throughput, latency, and error rate.

[0159] Furthermore, the transaction ID traceability module is embedded in Kafka;

[0160] The transaction ID traceability module is used to mark data change events with globally unique serial numbers and implement end-to-end exactly-once semantics by comparing them with version snapshots of the target database.

[0161] Based on the same inventive concept as the above disclosure, the present disclosure also provides an electronic device. The electronic device of the present disclosure embodiment includes at least one processor and at least one memory electrically connected to each other, the memory being electrically connected to the processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0162] It should be noted that the electrical connection between the above-mentioned units does not necessarily mean the connection between lines. An indirect connection method can be applied to the embodiments of the present disclosure as long as the purpose of the present disclosure is achieved.

[0163] Based on the same inventive concept, the present disclosure further provides a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0164] Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A big data update method based on Kafka and Flink CDC, characterized in that: include: Step S1: The configuration module receives the data source environment configuration and establishes a connection channel: receiving the data information defined by the user through the configuration template; Parse and register data information and establish a connection channel to the source database; the data information includes: source database type and connection parameters, target database type and address, and type definitions of each data table; Step S2: Flink CDC receives the real-time data change stream from the data source and obtains changes to the master table and sub-tables through pre-configured capture components. It formats the changed data to generate master table update messages and sub-table update messages. Step S3: The Kafka topic receives the master table update message and writes it to a pre-established master table-specific Kafka topic for storage and transmission, resulting in an asynchronous message queue for master table data changes. The master table-specific Kafka topic adopts a multi-partition high-throughput strategy. The topic also receives the sub-table update message and writes it to a pre-established sub-table-specific Kafka topic for storage and transmission, resulting in an asynchronous message queue for sub-table data changes. The sub-table-specific Kafka topic adopts a compact storage strategy. Step S4: The Flink job management module processes the asynchronous message queues for master table data changes and the asynchronous message queues for sub-table data changes to obtain processing results. Step S5: The target database updates the database using the processing results through the established connection channel; The target database includes multiple target database shards; After step S4 and before step S5, the method further includes: the Flink job management module uses a window sharding algorithm to divide data batches according to event time windows, and shards the high-priority task execution results and the low-priority task execution results based on the primary key hash value, wherein the number of data shards is equal to the number of shards of the target database; The step S5 specifically includes: Each target database shard receives the high-priority task execution result or the low-priority task execution result, as well as the pre-commit instruction issued by the two-phase commit coordinator, writes the data update operation into its own transaction buffer for pre-commit processing, and obtains the pre-commit execution result; The two-phase commit coordinator receives the pre-commit results returned by each target database shard. When it determines that the pre-commit of all shards has been successfully completed, it sends a global commit instruction to all target database shards to commit the transaction.

2. The method according to claim 1, characterized in that The data information also includes: the priority configuration corresponding to each data table; the Flink job management module includes: the Flink JobManager priority weight calculation module, the high-priority task thread pool, and the low-priority task queue; Step S4 specifically includes: Step S41: The Flink JobManager receives the main table update message and sub-table update message from the Kafka topic, adds the main table update message and sub-table update message to the task queue for real-time processing, and obtains a list of update tasks to be scheduled based on the source table type and preset priority attributes. Step S42: The priority weight calculation module calculates the corresponding priority weight value according to the table type, data freshness and business tag of each task, and adds the priority weight value to the task metadata to obtain a weighted task queue; Step S43: The resource scheduling module receives the weighted task queue, sorts it according to its weight value, and performs a diversion process: high-priority tasks with high weight are extracted from the queue and assigned to the high-priority dedicated channel, while low-priority tasks with low weight are retained in the ordinary channel for processing, thus obtaining two types of task sequences distinguished by priority. Step S44: The high-priority task thread pool receives the high-priority task and submits it to the dedicated thread pool for parallel execution, processing the key data update operation at high speed and obtaining the high-priority task execution result in real time; Step S45: The low-priority task queue receives low-priority tasks, adds them to the shared waiting queue, and queues them for processing in a predetermined order. These tasks are executed sequentially when the system is idle or under low load, and the low-priority task execution results are obtained in sequence.

3. The method according to claim 1, characterized in that Also includes: Step S6: The monitoring platform collects task indicators in real time and displays them dynamically on the dashboard; Failed tasks are automatically transferred to HDFS and trigger exponential backoff retries. Externally triggered exponential backoff retries are also accepted. Task metrics include some or all of throughput, latency, and error rate.

4. The method according to claim 1, wherein A transaction ID traceability module is embedded in Kafka; the method also includes: The transaction ID traceability module marks data change events with a globally unique serial number and implements end-to-end Exactly-Once semantics by comparing the target database's version snapshots.

5. A big data update system based on Kafka and Flink CDC, characterized by: include: Configuration module, FlinkCDC, Kafka Topic, Flink job management module and target database; The configuration module is used to receive data source environment configuration and establish a connection channel: receiving data information defined by the user through the configuration template; Parse and register data information and establish a connection channel to the source database; the data information includes: source database type and connection parameters, target database type and address, and type definitions of each data table; Flink CDC receives real-time data change streams from data sources and uses pre-configured capture components to capture changes to the master table and sub-tables. It then formats the changed data to generate master table update messages and sub-table update messages. Kafka topic, which receives master table update messages and writes them to a pre-established master table-specific Kafka topic for storage and transmission, resulting in an asynchronous message queue for master table data changes. This master table-specific Kafka topic uses a multi-partition high-throughput strategy. It also receives sub-table update messages and writes them to a pre-established sub-table-specific Kafka topic for storage and transmission, resulting in an asynchronous message queue for sub-table data changes. This sub-table-specific Kafka topic uses a compact storage strategy. The Flink job management module processes the asynchronous message queues for master table data changes and sub-table data changes to obtain processing results. The target database is used to update the database with the processing results through the established connection channel; The system further includes: a two-phase commit coordinator; the target database includes multiple target database shards; The Flink job management module also uses a window sharding algorithm to divide data batches into event time windows. It also uses the primary key hash value to shard the results of high-priority and low-priority tasks. The number of data shards is equal to the number of shards in the target database. The target database shard receives the high-priority task execution results or low-priority task execution results, as well as the pre-commit instructions issued by the two-phase commit coordinator, writes the data update operation into its own transaction buffer for pre-commit processing, and obtains the pre-commit execution results; The two-phase commit coordinator is used to receive the pre-commit results returned by each target database shard. When it is determined that the pre-commit of all shards has been successfully completed, it sends a global commit instruction to all target database shards to commit the transaction.

6. The system according to claim 5, characterized in that The data information also includes: the priority configuration corresponding to each data table; the system also includes: a resource scheduling module; the Flink job management module includes: a Flink JobManager priority weight calculation module, a high-priority task thread pool, and a low-priority task queue; The Flink JobManager receives primary and sub-table update messages from Kafka topics, adds them to the task queue for real-time processing, and obtains a list of update tasks to be scheduled based on their source table type and preset priority attributes. The priority weight calculation module is used to calculate the corresponding priority weight value according to the table type, data freshness and business tag of each task, and append the priority weight value to the task metadata to obtain a weighted task queue; The resource scheduling module is used to receive the weighted task queue, sort them according to their weight values, and perform diversion processing: high-priority tasks with high weights are extracted from the queue and assigned to high-priority dedicated channels, while low-priority tasks with low weights are retained in the ordinary channel waiting to be processed, resulting in two types of task sequences distinguished by priority; The high-priority task thread pool is used to receive high-priority tasks and submit them to a dedicated thread pool for parallel execution, processing key data update operations at high speed and obtaining high-priority task execution results in real time; The low-priority task queue is used to receive low-priority tasks, add them to the shared waiting queue and queue them for processing in a predetermined order, and execute these tasks in sequence when the system is idle or under low load, to obtain the low-priority task execution results processed sequentially.

7. The system according to claim 5, characterized in that Also includes: A monitoring platform, used to collect task indicators in real time and display them dynamically through dashboards; Failed tasks are automatically transferred to HDFS and trigger exponential backoff retries. Externally triggered exponential backoff retries are also accepted. Task metrics include some or all of throughput, latency, and error rate.

8. The system according to claim 5, wherein: Also includes: Transaction ID traceability module embedded in Kafka; The transaction ID traceability module is used to mark data change events with globally unique serial numbers and implement end-to-end exactly-once semantics by comparing them with version snapshots of the target database.

Images